Failure to save data for some websites

[vringar]

taehyung222 said in Matrix:

Hi, I observed that among the sites (from a list) that do not show an output in OpenWPM, a section of them do show an output if I query for just that one website. For example ("http://zhanqi.tv", "http://rednet.cn", "http://shutterstock.com", "http://cnnic.cn", "http://skype.com", "https://typeform.com", "http://cdc.gov").

I would be grateful to know on what could be done to be able to get output from a greater proportion of the sites in a given list.

However when running just these sites in isolation I seemed to be getting structured data for them. This needs further investigation.

This we need to know about:

• Which version were you running
• Which plattform are you running on
• Do you see any errors in the logs that could point us towards the error cause
• What kind of data are you missing? Structured or Unstructured or both

[shivani-1521]

OpenWPM version : v0.13.0 On Ubuntu 20.04.1 LTS with Python 3.8.5 and Conda 4.9.2 However, it appears this could be a false alarm as there were timeout errors, which could be due to my network.

[vringar]

Okay, v0.13.0 didn't have any changes to the data storing. Timeouts are unfortunately very common when running crawls. Consider having a look at openwpm-utils for analysis tools we use when analyzing our crawls