Update readme

[SebastianZimmeck]

Currently, the readme seems outdated as we decided to use the cloud crawler. @dadak-dom, can you add the cloud crawl instructions and remove everything outdated? @danielgoldelman can help with any questions.

[dadak-dom]

Readme has been updated 👍

[SebastianZimmeck]

There are a number of points that are not clear to me or that otherwise need to be fixed (e.g., formatting). I do not think anybody outside of our team can look at this and set this up or even understand what is going on. @dadak-dom, take a look at other readmes for examples, e.g., Privacy Pioneer or the GPC Web Crawler, Here are the main points:

• [x] Add internal links to sections
• [x] "Note: Due to problems with detecting location requests ... ." This is basically incomprehensible for anyone who was not part of our discussion. Better a short sentence or two along the lines of "Our crawler is intended to be used in different geographical locations as privacy laws may differ. To that end, we tested the use of a VPN, Web proxy, and cloud-bases solution. The latter proved to be the most effective and the following instructions are based on Google Cloud (link)." If there is more to say, open up a known issues sections at the end, and include there.
• [x] "In the same vein, we have resorted to hard-coding the expected location values in the extension." Basically incomprehensible for someone not in the know. People do not know about the API token, not even about the connection to Privacy Pioneer. This should be said, and also not straight in the beginning.
• [x] Does this repo need to be cloned? If so, say so.
• [x] "Instructions for how to do this are located at the bottom of the README." Include internal link.
• [x] "Before you can set up on the cloud, you need to create a virtual machine." On my local computer? On Google Cloud? Do I need a Google Cloud account? Unclear.
• [x] "Go to the privacy-pioneer-web-crawler project (you will need access to this)" and the following instructions are unclear as well. "access to this" means access to what? I need write access to the privacy-pioneer-web-crawler repo here or what is the "project"?
• [x] If this is on Google Cloud, include some screenshots of the most salient steps.
• [x] "2.1 Instructions for Setting up MySQL on macOS" Why is this included? Didn't we say Google Cloud only works on Windows machines? (Which, if that is the case, should be noted and explained in a know issues section as that is a pretty big limitation; then remove the macOS section).
• [x] "2.2 Instructions for Setting up MySQL on Windows Install MySQL." Link and/or explanation? Maybe reference to the GPC Web Crawler repo Wik?
• [x] "Enter your root password." This is a password on Windows or the DB password below? If the latter, this is out of order. Windows people will not have done the macOS setup.
• [x] Also, why am I doing this locally when we are using the Cloud? Include an explanation of the general architecture (in a section at the beginning?)
• [x] "Install MySQL and the MySQL shell." How? Link and brief explanation.
• [x] "CREATE TABLE entries ..." format in multiple lines.
• [x] "DB_USERNAME=root" Does this work without quotes, e.g., "root"?
• [x] "If using Windows, navigate to ... " This is still in the SQL shell or do I need some sort of UNIX command line interface here on Windows?
• [x] "These two commands are enough to get the crawl to run. You should now see ... ." How do I know it is working?
• [x] "Here are the steps for creating a new version of the extension (this is important because each location will have a different version)" Grammar! Please include periods. Also, "It is advantageous to create a separate crawl instance for each geographical location you want to crawl because ... ."
• [x] "In the privacy-pioneer source code, navigate to src/background/analysis/buildUserData/importSearchDarta.js" Link to file.
• [ ] "Above line 77 (let locKey = ....), set locCoords = [lat, long]" Link to specific line of code. Do not include the number as it may become 78 etc.
• [x] Fix formatting for "npm run build" as that should be code with a copy mechanism.
• [x] "Change the file from .zip to .xpi (just rename it)" You always want to briefly say "why" and not just "how." This applies to code comments as well. Otherwise, people may find it a mystery why they should do something, how code is connected to other code, etc.
• [x] "At this point, see issue 541 in PP." Unclear. No idea what this refers to. Please try to find it out and fix.

Please take another pass @dadak-dom and spend some time to make the readme good. If there is anyything that you do not understand, clarify it or remove it. If there is anything that is outdated, it should be removed. Imagine you know nothing about Privacy Pioneer and you just found this code because you searched "data collection and sharing open source code" or something like that, and now you try to make sense of that and set this up. What do you need to know?

[dadak-dom]

Got it, will work on fixes. Thank you for the feedback @SebastianZimmeck 👍

[SebastianZimmeck]

Great! Thank you, @dadak-dom!

[SebastianZimmeck]

@natelevinson10 will also check if the updated readme is understandable.

[SebastianZimmeck]

Another point for the readme.