mxsnq
commented
4 years ago What I did:
- Refactored output format of selectors. Text, attribute, style, html selectors now return lists of values. Image/doc selectors now return lists of URLs and lists of attachment objects. Data extractor collects all media attachments of an item under _attachments field and adds url and timestamp. Improved code for other selectors.
- Updated export data form. Added export to JSON lines. Updated CSV export so it works like data format has not changed.
Some notes and future work:
- PouchDB has its own format for attachments, which differs from ours. Also it actually forbids any keys starting with underscore besides _id, _rev and _attachments. So I decided to wrap crawled items in another objects on saving and unwrap them back on loading from DB. It looks like this
{ data: { [selector-id]: "its value", // or list of values ... _url: https://example.com/blahblah, _timestamp: ..., _attachments: [...] } }Proposal for next version: I believe we could move our "_hidden" properties one level up, so we don't mix them with selectors
{ url: https://example.com/blahblah, timestamp: ..., attachments: [...], // or _attachments in PouchDB format data: { [selector-id]: "its value", // or list of values ... } }Reminds me of format we have used for some social networks.
- There is still lots of room for improvements in the code. Third of selectors follows the same patttern get elements > map to data > check if we should return list or single value > return, which may be generalized in base Selector. Text manipulations should be refactored and extracted to separate class. Same goes to data exporters. I almost did not change DataExtractor. Some data processing functions may be simplified knowing that we can't have nested objects in data. And more... I'd like to continue my work on these.
- I designed new extract data form similar to old version: same inputs, button to generate export file, then link-button to download. I'd like to reimplement this with single button, and place some loading animation instead of "Please, wait" alert.
- I have encountered some weird error with sitemaps creation. When one submits create sitemap form with untouched model input, it somehow crashes and reloads devtools panel. I managed to mitigate that by removing type="submit" from button, so bootstrapValidator does not submit the form - seems like it does so automatically when it finds out all fields have been validated. But I still don't understand how it works and why edit metadata form works fine without such adjustment. Very weird.
Closes #27.