Errors in push surveyor_sheets*

[dbl3raf]

I've tried all troubleshooting I can think of:

[jeffreyhanson]

Hmm, I thought i fixed this error in an earlier commit - do you have the latest version of the repo code?

[dbl3raf]

yep I've synced via Github desktop - trying the push again now

[dbl3raf]

still get the same error on trying to push assets

[Louis-Backstrom]

does everyone get this error or is it just @dbl3raf ?

[Louis-Backstrom]

Googling this error indicates that a likely culprit is something to do with a date format - do we have any dates in the new dataset that might be "ambiguous" or something? Can we trace this problem back to a commit someone made?

[jeffreyhanson]

Hmm, my computer shows the same error when I try to pull the assets - I'll see what I can do.

[jeffreyhanson]

I've tried updating the version of the piggyback R package in the build environment, so once the docker image has finished being rebuilt we can see if that fixes it (https://hub.docker.com/r/brisbanebirdteam/build-env/builds).

[jeffreyhanson]

I've updated the R package responsible for interacting with GitHub (i.e. piggyback), can you please update your docker image and try again @dbl3raf? You can update your docker image with the commands:

make rm_image
make pull_image

[dbl3raf]

Definitely fixed that date format error (assets build just fine), but I'm getting the old errors on pushing the assets again. I've tried twice tonight, and will have another go tomorrow.

[dbl3raf]

happens on different survey sheets each time

[dbl3raf]

3 more push failures of the same type. I tried deleting all assets from the release.

[jeffreyhanson]

I just had a quick look at the releases and noticed the one of the pdfs had a red cross next to it - could you try deleting that and rerunning it?

[jeffreyhanson]

IIRC you were experiencing the same issue last week. How did you resolve it? Was it that you needed to run the code at UQ (e.g. for better internet connection)?

[jeffreyhanson]

We might have to look into hosting the surveyor sheets somewhere else since it looks like pushing ~500 assets to GitHub is a bit unreliable.

[dbl3raf]

I've always been deleting the sheet with the red cross, and all the zip files between attempts to push assets. I also tried deleting every single surveyor sheet and all the zip files - still eventually fails on one of the uploads. Resolution last time was just a fluke - it just happened to work, but I agree it it appears that this success isn't reliably repeatable

[dbl3raf]

is there any way to direct upload the pdfs into the release via the web interface?

[jeffreyhanson]

Ah, ok. Yeah, if you (1) click the button to edit the release, (2) remove all the files in the release, (3) select all the new files in the file explorer on your computer, (4) drag them on to the grey box that says "Attach binaries by dropping them here or selecting them", (5) wait for them all to upload, and then (6) click "Update release".

[dbl3raf]

silly question but it appears the only way to delete files from the release is by clicking on every single cross - is that right?

[jeffreyhanson]

Yeah, that's correct (unfortunately). If the drag and drop approach works - I'll add a command to the Makefile to delete the assets so you can easily drag + drop the assets at will.

[dbl3raf]

This has worked - I uploaded all the surveyor sheets directly into the release, and the drag and drop worked perfectly even for 500+ files. Perhaps this is the easiest way to push the assets - it would be great @jeffreyhanson if you could add a function to delete existing assets so they can be quickly replaced using drag and drop.

[jeffreyhanson]

Ok - I could do that but I don't like of having a "manual" step in the process. I feel like atlas admins should be able to enter the command make push_assets and just have it work. What do you think? But to do this it looks like we might have to abandon GitHub releases for the surveyor sheets and use something else like box.com or Amazon S3.

[jeffreyhanson]

I've created an issue on the piggyback R package GitHub repo to see if the developer has any ideas for fixing this (https://github.com/ropensci/piggyback/issues/19).

[jeffreyhanson]

As a hotfix, I've just added the command make rm_gh_surveyor_sheets which will delete all the files in the surveyor sheet release (but this won't delete any files that have the red cross problem, and these will still need to be deleted manually).

Also, I deleted all the surveyor sheets when testing the code, so if you could please re-upload them that would be great!

[jeffreyhanson]

@dbl3raf, we've updated the piggyback R package so that it can re-upload files when they fail to upload correctly, and I've rebuilt the docker image with the new version of the piggyback R package. Can you fetch the latest copy of the docker image, and try uploading the assets again? You could do this with these commands:

make rm_image
make pull_image
make push_assets

[dbl3raf]

removed and pulled the new image - but same old error on attempting to push assets:

[jeffreyhanson]

I've just pushed another couple of changes to the BBA code for pushing the surveyor sheets - can you pull the latest version of the repo and try pushing again with make push_assets?

[dbl3raf]

make push_assets still fails on surveyor sheets. This is OK - I'm fine with deleting and manually uploading the sheets, but unfortunately the command make rm_gh_surveyor_sheets isn't working either. It reports completing, but the surveyor sheets are still in the release once it's done, and have to be manually removed.

[jeffreyhanson]

Hmm, could you please copy and paste the output that you get when running it?

[dbl3raf]

[dbl3raf]

and here's a shot of the release straight after running the code

[jeffreyhanson]

Sorry for the delay, I was able to replicate this on my computer and the fix was to use the latest version of the piggyback R package. So, I've updated the code for the atlas build environment (https://github.com/bird-team/build-env). Once DockerHub has finished building the atlas image (see https://hub.docker.com/r/brisbanebirdteam/build-env/builds, this will probably take around an hour or so, you should see "Success" on the latest build when its done), you need to update your version of the build environment (with make rm_image pull_image). Then you should be able to delete the surveyor sheets at will with make rm_gh_surveyor_sheets. Also, as part of testing the fix, I ended up deleting all the surveyor sheets, so could you please reupload them?

[Louis-Backstrom]

Hopefully my complete rebuild of the assets (happening now) will fix this. Will update.

[jeffreyhanson]

Ok - yeah hopefully, this fixes it - thanks.

[Louis-Backstrom]

No luck sadly, make push_assets still failed halfway through the surveyor sheets.

I ran make rm_image and make pull_image before starting the build, and I was going off a clean download of the repo, so not sure what the issue is.

[jeffreyhanson]

Yeah, I don't know. I think GitHub's API for uploading and downloading files can be a bit flaky at times. Could you please try manually uploading the files to the surveyor sheets release? I'm sorry, I don't have any ideas on how to fix this at the moment.

[Louis-Backstrom]

Yeah, that's what I've done. Could we perhaps split up the make push_assets step into two, e.g. make push_assets and make push_sheets to split up the load?

Or even just remove the surveyor sheets from the push at all and let the user upload them manually?

[jeffreyhanson]

Yeah, that's a great idea. I'll try and remove the sheets from the make push_assets command and make a new make push_sheets command tomorrow if I have time. I think it's still important to try and support the automated surveyor sheet pushing - I would hope that in time the GItHub API becomes more stable such that it should work without any changes on our end.

[Louis-Backstrom]

Great, thanks @jeffreyhanson

[jeffreyhanson]

Ok, I've just pushed an update to (i) remove the surveyor sheet pushing from the make push_assets command, and (ii) add a separate make push_surveyor_sheets command dedicated for (failing at the moment) pushing the surveyor sheets.

[Louis-Backstrom]

great, thanks @jeffreyhanson - I don't have much time the rest of this week do be test building but I'll run this change when / if I get the chance and see how it does.

[jeffreyhanson]

Ok - no rush - thank you.

[Louis-Backstrom]

Still no good with make push_surveyor_sheets - it got to about halfway through overnight then crashed. Manually uploading the sheets again now. Could we perhaps try splitting the command up into 5 or so subcommands so it uploads 100 at a time? If the issue is unreliability that could bypass that, seeing as it doesn't seem to fail on any given sheet it just fails (presumably) when something goes wrong with the internet.

Or could we make the make push_surveyor_sheets command somehow retry a couple of times if it fails on a sheet?

[jeffreyhanson]

Yeah the code for pushing the surveyor sheets involves pushing each sheet individually, and already tries to detect when it has failed and retries 20 times, waiting five seconds between each try, before giving up. I could try increasing the counter to something like 1000?

[Louis-Backstrom]

Hmm okay that's weird then.

Also, even when it works properly (I had it get at least 2/3 of the way through last time before I killed it) it's so much slower than just manually doing it via the website - is there a way to fix / alleviate this?