Open emigre459 opened 4 weeks ago
Can't figure out why, but found that containerizing the selenium code causes it to hang indefinitely after it exits the login dialog on Plugshare. Abandoning the container approach for a messier "just run it on a GCP VM after installing what you need" approach to get things moving.
VM spun up
The theory is that we'll be able to run our scripts faster on a dedicated server with high-speed networking...we shall see!
h3_hexagons_usa.pkl
(and IDs from index) data we use into BigQuery for hopefully faster referencing (e.g. using an OFFSET call per scraping worker)