data from JS pages is not returned

[chaitra-rs]

Trying to get data from a page which has JS scripts. Page shows data but output file doesn't.

Please describe our issue, along with:

• Output file should have all data present in page
• Some data missed out in output file.

How to reproduce it

Try injecting the page https://www.ibm.com/support/pages/node/884036 Data in Document Information will be missed out in part-00000 file

Environment and Version Information

Please indicate relevant versions, including, if relevant:

Followed the steps in the document (used docker) shows sparkler@5c224e262222:/$

[thammegowda]

@chaitra-rs thanks for reporting this. By default, the javascript execution is not enabled (why? JS execution is very very slow)

However, if you do need it, you need to enable a plugin.

https://github.com/USCDataScience/sparkler/blob/5c2201310623b70e6bf024e51e521eb4bffc4723/conf/sparkler-default.yaml#L102-L104

how? locate sparkler-default.yamlinside the docker that is being used by sparkler, and uncomment one of those fetcher plugins capable of executing javascript.

I dont know which one is best since each has their +s and -s (suggest trial and error for your usecase.).