add crawl lists - Githubissues

privacy-tech-lab / privacy-pioneer-web-crawler

Web crawler for detecting websites' data collection and sharing practices at scale using Privacy Pioneer

https://privacytechlab.org/

MIT License

0 stars 0 forks source link

add crawl lists #11

Closed dadak-dom closed 5 months ago

SebastianZimmeck commented 6 months ago

A few questions:

What protocol did you use to create the lists?
Do we have the original lists and metadata (e.g., date of the original list) from which the lists were created?
In the Australia list line 455, https://becoolrefrigeratedcouriers.com is not an Australian top-level domain site. Please check that we have the correct domain for all sites in all lists.
When I visit https://pmc.gov.au, I get redirected to https://www.pmc.gov.au/. What does our crawler do in such a situation?
Are we sure that Ukrainian, Portuguese, and other non-English sites are properly analyzed (especially, for location data types and keywords)? See @JoeChampeau's comment.

dadak-dom commented 6 months ago

@SebastianZimmeck To answer some of your questions:

I made a Google Doc in the Privacy Pioneer folder that outlines the steps for creating a list, as well as our motivations for each country. It can be found here
Yes, the original tranco list is one of the lists that were added to this branch. It's the list that was made available on Nov. 27th, 2023.
That's strange, that site shouldn't be there... I'll do another pass on the lists at some point and make sure that they're all good. That one must have slipped through somehow.
Daniel addressed this point in our last meeting, that such redirects are not a problem for the crawler
We will be looking into keywords over the coming days

SebastianZimmeck commented 6 months ago

Thanks @dadak-dom!

Yes, the original tranco list is one of the lists that were added to this branch. It's the list that was made available on Nov. 27th, 2023.

This is the list, I assume?

dadak-dom commented 6 months ago

Yep, that's the list!

danielgoldelman commented 5 months ago

@dadak-dom Are there any other tasks still to do on this pr?

dadak-dom commented 5 months ago

@danielgoldelman As far as I knows it should be all good 👍

SebastianZimmeck commented 5 months ago

Are the above points resolved?

In the Australia list line 455, https://becoolrefrigeratedcouriers.com/ is not an Australian top-level domain site. Please check that we have the correct domain for all sites in all lists.

Are we sure that Ukrainian, Portuguese, and other non-English sites are properly analyzed (especially, for location data types and keywords)? See https://github.com/privacy-tech-lab/privacy-pioneer-web-crawler/issues/7#issuecomment-1870984369.

dadak-dom commented 5 months ago

@SebastianZimmeck I've fixed the issue with the list, but the analysis is still in progress.