andykais / scrape-pages

generalized scraper using a single instruction set for any site that can be statically scraped
https://scrape-pages.js.org
MIT License
6 stars 2 forks source link

create query function before creating folders/tables #15

Closed andykais closed 5 years ago

andykais commented 5 years ago

this is the ideal flow we want to create:

const { start, query } = scrape(config, options, params)

const { on, emit } = await start()
const result = query({ scrapers: ['image'] })

This reduces the surface api in cases where we want to create a query function separate from the execution. E.g. a worker thread is running while a parent thread is querying.

it is achievable, we just need to create the query function without creating tables & preparing statements.

andykais commented 5 years ago

closed in #8