search

bird-team / brisbane-bird-atlas

Atlas of the Birds of Brisbane: Community bird atlas for Brisbane, Australia
https://brisbanebirds.com
GNU General Public License v3.0
3 stars 0 forks source link

Trouble remaking the assets #119

Closed Louis-Backstrom closed 5 years ago

Louis-Backstrom commented 5 years ago

I'm trying to update the assets myself (instead of letting @dbl3raf do it) to reflect the changes to the parameters.yaml made in eb41c71 (breeding codes). So far I have:

Cygwin installed Docker Desktop installed and running (little icon in quick start menu telling me it's running) .Renviron file with the correct passcode in C;\Users\louis\Documents.Renviron and C;\Users\louis.Renviron (as Rich and I were unclear which one it's supposed to be in)

Opened up a cmd window and navigated the directory to the GitHub\brisbane-bird-atlas folder

But when I execute make pull_image as the FAQ tells me to do, I get this error:

image

Various fiddling around with variations on this also don't seem to work. I'm at a dead end - @jeffreyhanson ?

jeffreyhanson commented 5 years ago

Great! That's strange, could you please try installing the latest official version of Rtools (https://cran.r-project.org/bin/windows/Rtools/, version 35 I think)? When installing this, please install it under admin privileges and make sure the option is checked to add it to your system's PATH environmental variable.

Louis-Backstrom commented 5 years ago

Fantastic, I got much further along now.

image

This is the error I'm getting at the moment - my .Renviron file only has one line (the GITHUB_PAT string) - I'm assuming I need another one?

Also, is it safe to edit the parameters file to change the threads at any stage during the process (provided nothing is currently running) or should I do it before I start it at all?

jeffreyhanson commented 5 years ago

Yeah, you need a "GOOGLE_TOKEN" string. Unfortunately, to download google maps, you need to create a google developer account, and generate an key for accessing their maps API. Google will also link this developer account to a payment system (e.g. credit card, bank account). In addition to the maps API, the developer account can access APIs for cloud compute systems and other costly services so you need to keep this API key safe. IIRC, you can download 10k google maps per month before getting charged - I haven't been charged yet. For instructions, see here (note this isn't an official guide, so I wouldn't click any links in case of spoofing, but instead manually type out the URLs, but perhaps I am more careful/paranoid then others). After you get the key, open the .Renviron file and add a new line which reads GOOGLE_TOKEN=XXXXXXXXXXX where XXXXXXXXXXX is your API key, save and close the file, and then try rerunning make assets.

Yeah, if it crashes after a certain point, you can edit the threads parameter, but don't edit it while something is running (best case scenario, nothing happens, worst case scenario, not sure, probably nothing happens).

Louis-Backstrom commented 5 years ago

Great, thanks @jeffreyhanson

Now I'm getting a new error: image

I have a vague feeling I remember seeing this error before on an issue or with @dbl3raf but I can't relocate any evidence of that. Any ideas?

jeffreyhanson commented 5 years ago

Ok great - hmm, it might be that your computer is running out of RAM? Maybe try closing all other programs and also reducing the number of threads?

jeffreyhanson commented 5 years ago

Or, it might be that docker has imposed a certain limit on the amount of RAM it can access? Maybe try opening the docker graphical user interface program and see if you can change the maximum amount of RAM it can use?

Louis-Backstrom commented 5 years ago

Okay, it failed when I ran it on just the 1 thread with everything else closed - RAM usage didn't get over 8GB (I have 16) and CPU was cruisy the whole time. I have just changed docker to use 5 CPUs (/6) instead of 1 (I wonder, I had the paramters.yaml file variously set to use 1, 2 and 5 threads at different attempts, maybe it crashes if parameters file is larger than the number docker gets?) and I've allocated 8GB of RAM instead of 2. Will try again.

E: I think we're good now :) Will update if it crashes again but I'm hopeful. Will let it run through on 1 thread and then I'll try it with 5 to see if it's faster.

jeffreyhanson commented 5 years ago

Brilliant! Yeah, please let me know if it crashes again.

Louis-Backstrom commented 5 years ago

Many thanks @jeffreyhanson - you're a wizard. I've gone and edited the FAQ section on all of this to hopefully make it a bit easier to follow / DIY, and the code is still running happily. At this rate it looks like it'll take a while (only up to surveyor sheet 9925) but hopefully I can re-run it tomorrow morning with more threads and cut down the time.

jeffreyhanson commented 5 years ago

No worries - thank you for persisting with this! Awesome - thanks for updating the FAQ! Yeah, it can take a long time to finish making all the assets from scratch given the sheer size of the data.

Louis-Backstrom commented 5 years ago

image

Pushing assets to storage failed at some stage during the day (only just got home to see it) - I've got a GITHUB-PAT set in the .Renviron - should there be a separate GITHUB_TOKEN?

jeffreyhanson commented 5 years ago

Yeah, just copy the GITHUB_PAT token line, and rename the copy as GITHUB_TOKEN (IIRC you need both).

Louis-Backstrom commented 5 years ago

What a weird idiosyncracy... I'll see how that goes now.

jeffreyhanson commented 5 years ago

Yeah, I agree it's a bit odd.

Louis-Backstrom commented 5 years ago

image

Same error overnight - seems to fail around the same time (10140s)?

Louis-Backstrom commented 5 years ago

Somewhat confusingly, the graphs seem to have updated (to the best of my knowledge - for instance, the Crested Pigeon has breeding data in September, which is only of CN, which wasn't previously one of the codes used for the breeding graph but now is) - I think it's probably just failed to push the last 400 or so surveyor sheets, which is a problem in the sense that it should be able to, but isn't in the sense that I don't think they have changed as I've only edited the breeding data.

Edit: Only the surveyor sheets that were successful are able to be downloaded - the rest just give the Not Found page. So it would be good to fix.

jeffreyhanson commented 5 years ago

Hmm, that's strange. The code assumes that you have constant internet access during the entire upload process. Is it possible that your internet connection died briefly over night? Could you please try this again? I suppose one potential fix could be to try and detect if the already pdfs on GitHub are identical to that on your computer, and only upload pdfs which are different. This could be rather tricky though.

jeffreyhanson commented 5 years ago

Now that I think about it, @dbl3raf was also having problems using the upload functionality and had (or is still?) to upload surveyor sheets by dragging and dropping them on GitHub. Did you find a fix @dbl3raf?

Louis-Backstrom commented 5 years ago

I was having internet troubles actually yeah, although I thought they started a bit after - maybe not. I think I'll try remaking all the assets again (to fix the graphs and surveyor sheets) and see how it goes - might be a day or so before we can be sure.

Louis-Backstrom commented 5 years ago

image I'm thoroughly confused by this error... Will retry make assets again and see if it sorts itself out.

E: Can't get past this error. Have tried making assets completely from scratch, restarting docker, etc. Resource usage seems normal the entire time as well so I can't see it being that - Docker has 5 cores and 8GB of RAM allocated.

jeffreyhanson commented 5 years ago

Hmm, yeah that's odd. The text after Killed are just useless warning messages that we can ignore. Generally, the Killed error message means that it has run out of RAM. Could you please try increasing the amount of RAM allocated to Docker if possible? If not, could you please try decreasing the amount of threads set in the parameters YAML file?

jeffreyhanson commented 5 years ago

Also, I think this is unrelated to the patch I pushed to try and address #114. If the patch had a bug in it, we would be seeing an error when it tries making the graphs but it looks like it's crashing when trying to make the surveyor sheets.

Louis-Backstrom commented 5 years ago

Hmm, yeah that's odd. The text after Killed are just useless warning messages that we can ignore. Generally, the Killed error message means that it has run out of RAM. Could you please try increasing the amount of RAM allocated to Docker if possible? If not, could you please try decreasing the amount of threads set in the parameters YAML file?

I set it running this morning with 10GB (really pushing my computer) and 5 cores allocated, but only 2 threads in the parameters file. It was happy when I left it a couple of hours ago so hopefully it's fine. It was giving me lots and lots of warnings about the nonempty <title> element> though, which I don't remember having before.

Will be home in 6 hours to see if it's sorted itself out successfully.

Louis-Backstrom commented 5 years ago

image So the build itself was successful today but uploading again is giving me the same grief as before. Will see if I can work out how to manually upload them (as it seems that the assets themselves uploaded fine).

Louis-Backstrom commented 5 years ago

Right, surveyor sheets successfully uploaded. Going to completely wipe the atlas repo from my computer, redownload it from GitHub and create all the assets from scratch and see what fun things happen this time. Hopefully it will address the fix in #114

Louis-Backstrom commented 5 years ago

Build overnight went off without a hitch, but still having problems with the surveyor sheet push. I'm going to close this issue for now as the assets themselves were fine, and redirect all my problems with the push itself to #115.

jeffreyhanson commented 5 years ago

Ok - great to hear the build and assets are now working.