Closed andykais closed 5 years ago
theres an important difference in the data flow. Before, data only flew RIGHT and DOWN (scrapeEach array and scrapeEach). Now, there is a third direction. "DIAGONAL-LEFT". E.g. data can hit a branch, then merge back into the downward stream. Its causing a pretty big change in the sql ordering so I want to know if its worth it. The real world use case is, we have multiple places where a single type of data needs to be downloaded/parsed. E.g. traverse several different pages to get all the image links on a site. It could be important to be able to grab all those scrapers at once with the querier.
What I do like about this: its a really intuitive syntax, easy to wrap your head around. E.g.
[
{
scraper: gallery,
branch: [[pageA], [pageB]]
},
downloadImage
]
what I dont like about this: the sql ordering isnt that smart. Before I could assume that I was dealing with a tree data structure. Now Im dealing with a one way graph. Its much harder to reason about order in this case
gallery - pageA
\ \
pageB - downloadImage
an alternative structure for the config, one that is more flat, less nested.