Closed Louis-Backstrom closed 3 years ago
Hi,
Thanks for looking into updating the atlas! I'm not exactly sure what's going wrong here, it seems like the command prompt on your system is unable to find Docker. Could you please confirm if this is the case by (i) opening the command prompt, and (ii) typing in the command docker version
, and (iii) pressing enter? You should then see something like this if the command prompt can find Docker on your system:
Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:49 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:44:20 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Client: Docker Engine - Community
Cloud integration: 1.0.4
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:14:16 2020
OS/Arch: windows/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:23 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Seems to think so?
Yeah - it looks like the command prompt can indeed find it. What happens if you run the command docker pull brisbanebirdteam/build-env:latest
?
C:\Users\louis>cd C:\Users\louis\Documents\GitHub\brisbane-bird-atlas
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>docker pull brisbanebirdteam/build-env:latest
latest: Pulling from brisbanebirdteam/build-env
ab1fc7e4bf91: Pull complete 4c10a1267cde: Pull complete ad73920d6196: Pull complete 97ad732d7102: Pull complete 50f553aef9eb: Pull complete b7e03bdd4db0: Pull complete Digest: sha256:16b26c8c59892465500f2b1e2ec87e0e67fbf692f66e05ea7dd18aeb7628c1ac
Status: Downloaded newer image for brisbanebirdteam/build-env:latest
docker.io/brisbanebirdteam/build-env:latest
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make pull_image
make: docker: Command not found
make: *** [Makefile:228: pull_image] Error 127
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make assets
sh: /ProgramData/DockerDesktop/version-bin/docker: /usr/bin/env: bad interpreter: No such file or directory
sh: /ProgramData/DockerDesktop/version-bin/docker: /usr/bin/env: bad interpreter: No such file or directory
sh: /ProgramData/DockerDesktop/version-bin/docker: /usr/bin/env: bad interpreter: No such file or directory
make: *** No rule to make target 'logos', needed by 'assets'. Stop.
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>
Seems to happily download the latest build environment but still no dice.
Sorry for the slow response. Do you remember if we installed make
on your computer using Rtools or Cygwin (or both? If you look for C:/cygwin
or C:/Rtools
/ or C:/rtools
on your computer that should tell you?)? If it was Rtools, maybe try uninstalling Rtools and reinstalling it? You can install the latest Rtools from here (https://cran.r-project.org/bin/windows/Rtools/rtools40-x86_64.exe, https://cran.r-project.org/bin/windows/Rtools/)
I have both Cygwin (x64) and Rtools (v40) on my computer. I've reinstalled Rtools recently and it didn't fix the problem.
Ah ok, what happens if (i) you open up the command prompt, (ii) enter the command sh
, and then (iii) enter the command docker version
? Note that it might take a few minutes for the command prompt to display something -- that's normal -- but hopefully you should see the same result as entering in docker version
into a fresh command prompt window?
C:\Users\louis>sh
sh-4.4$ docker version
sh: /ProgramData/DockerDesktop/version-bin/docker: /usr/bin/env: bad interpreter: No such file or directory
sh-4.4$
Hmm...
Hmm, I wonder what happens if you run docker version
in the Cygwin terminal? If you search for Cygwin in the start menu, there you should be a program with an icon that looks something like this?
$ docker version
Client: Docker Engine - Community
Cloud integration: 1.0.4
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:14:16 2020
OS/Arch: windows/amd64
Context: default
Experimental: true
error during connect: This error may indicate that the docker daemon is not runn
ing.: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/version: open //./pipe/doc
ker_engine: The system cannot find the file specified.
Then, with docker running:
$ docker version
Client: Docker Engine - Community
Cloud integration: 1.0.7
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:14:16 2020
OS/Arch: windows/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
(note: regardless of whether docker is running, the sh
command in cmd still fails)
Interesting! Thanks for checking if sh
still fails even if docker is running. Could you try running the make pull_image
command in the Cygwin terminal?
$ make pull_image
make: *** No rule to make target 'pull_image'. Stop.
Oh, you might need to change the working directory to the atlas folder first? On my computer, you would do something like this (in Cygwin) before entering the Makefile commands:
cd "C:/Users/jeffh/Documents/GitHub/brisbane-bird-atlas"
$ cd C:/Users/louis/Documents/GitHub/brisbane-bird-atlas
$ make pull_image
make: docker: Command not found
make: *** [Makefile:228: pull_image] Error 127
Ah - that's annoying - nearly thought we had a fix
Hmm, I might have to stew on this - I can't think of why this isn't working sorry.
Alright - I wonder if it is related to some other issues my computer is having at the moment. Thinking I might do a complete wipe and reinstall of windows at some stage in the next week, so if you don't have any other ideas I'll try that.
Yeah, I'm not really sure. I wouldn't bother wiping and reinstalling windows specifically because of this issue -- I'm sure there must be something else we could try before that as a last resort. But if you were planning on doing this anyway, it might be worth seeing if that fixes the issue?
Right. Computer totally hit the bucket a few days ago so I pulled the plug and did a fresh install (which appears to have fixed most of the problems). Still having issues with this though:
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make pull_image
'make' is not recognized as an internal or external command,
operable program or batch file.
When I run docker version
it's happy enough to show me everything, so it seems docker is installed (and it thinks it's running). Cygwin and Rtools are both installed.
My only current thought is something to do with the base architecture? For memory docker used to run on Hyper-V, but then Windows 10 updated at some point in the past two years to add Windows Subsystem for Linux, which I believe is what my current installation is using. Not sure why that would be an issue but that's all I have at the moment.
Ah, sorry to hear your computer died - glad that doing a fresh install fixed most of the issues.
Hmm, I think the issue might be that Rtools is missing from the PATH variable. If you check out the PATH variable (see https://helpdeskgeek.com/windows-10/add-windows-path-environment-variable/), can you check if the Rtools folder is added there (i.e. something like C:\rtools40\usr\bin
)? If not, can you add it and see if running make pull_image
in the command prompt works (note you will need to open a new command prompt after editing the PATH variable, and possibly restart the computer).
Good news - that worked!
Less good news, while make pull_assets
worked, make assets
is giving me an error:
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make assets
7292261a5824e48e05e06c171c5dee7407d58a3fa3ac1f7497f2fb2bcec9d1b1
adding: assets/ (stored 0%)
adding: assets/widgets/ (stored 0%)
adding: assets/widgets/.gitkeep (stored 0%)
adding: assets/tables/ (stored 0%)
adding: assets/tables/.gitkeep (stored 0%)
adding: assets/libs/ (stored 0%)
adding: assets/libs/player56s/ (stored 0%)
adding: assets/libs/player56s/player56s.min.js (deflated 80%)
adding: assets/libs/player56s/player56s_sprite.png (stored 0%)
adding: assets/libs/player56s/player56s.css (deflated 80%)
adding: assets/libs/player56s/player56s.js (deflated 80%)
adding: assets/profile/ (stored 0%)
adding: assets/profile/.gitkeep (stored 0%)
adding: assets/badges/ (stored 0%)
adding: assets/badges/iucn.svg (deflated 45%)
adding: assets/badges/.gitkeep (stored 0%)
adding: assets/badges/xenocanto.svg (deflated 49%)
adding: assets/badges/botw.svg (deflated 48%)
adding: assets/badges/ala.svg (deflated 43%)
adding: assets/badges/bla.svg (deflated 43%)
adding: assets/badges/ebird.svg (deflated 42%)
adding: assets/badges/bli.svg (deflated 40%)
adding: assets/badges/inaturalist.svg (deflated 47%)
adding: assets/maps/ (stored 0%)
adding: assets/maps/.gitkeep (stored 0%)
adding: assets/misc/ (stored 0%)
adding: assets/misc/.gitkeep (stored 0%)
adding: assets/misc/bsc.ai (deflated 11%)
adding: assets/misc/missing.png (deflated 6%)
adding: assets/misc/cover.ai (deflated 3%)
adding: assets/misc/test-badge.svg (deflated 43%)
adding: assets/misc/missing-profile.png (deflated 30%)
adding: assets/misc/cc.png (stored 0%)
adding: assets/misc/favicon.ico (deflated 9%)
adding: assets/misc/copyright.png (deflated 13%)
adding: assets/misc/book-cover.png (deflated 2%)
adding: assets/misc/eBird-profile.jpg (deflated 20%)
adding: assets/misc/bq-logo.png (deflated 1%)
bba
bba
make: *** No rule to make target 'logos', needed by 'assets'. Stop.
Awesome - looks like all the software is working together now!
Sorry - that's due to a typo in the Makefile. I've just pushed a fix to GitHub. Could you please pull the latest version from GitHub, and try running make assets
again?
Totally brilliant this is working! Louis and I will have to work out a strategy for reviving and pushing forward with the Atlas
From: Jeff Hanson notifications@github.com Sent: Thursday, 28 January 2021 3:27 PM To: bird-team/brisbane-bird-atlas brisbane-bird-atlas@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [bird-team/brisbane-bird-atlas] Trouble remaking assets (#153)
Awesome - looks like all the software is working together now!
Sorry - that's due to a typo in the Makefile. I've just pushed a fix to GitHub. Could you please pull the latest version from GitHub, and try running make assets again?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/bird-team/brisbane-bird-atlas/issues/153#issuecomment-768808654, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFTV4NCXJDNIMRW2IVYOOXTS4DYRHANCNFSM4WF5SL3A.
Alright, getting there now. Surveyor sheets have built correctly, now the graphs are acting funny:
starting graphs...
Anseranas semipalmata
Error in grid.newpage() :
could not open file 'assets/graphs/Anseranas-semipalmata.png'
Calls: <Anonymous> ... grid.draw.ggplot -> print -> print.ggassemble -> grid.newpage
Execution halted
bba
bba
Hmm, do you have this folder on your computer brisbane-bird-atlas/assets/graphs
?
I've just pushed a commit to add (mostly) empty assets/graphs
and assets/surveyor-sheets
folders to the repo. Could you try pulling the latest version from GitHub and re-running make assets
? If you run into any issues grabbing the latest version, you might need to delete some files in the brisbane-bird-atlas
folder so it matches the version on GitHub.
Right, assets built successfully once I did that. Now I'm just having trouble pushing them:
Starting to push assets to storage..
Error in value[[3L]](cond) :
Cannot access release data for repository bird-team/brisbane-bird-atlas. Check that you have set a GITHUB_TOKEN and that at least one release on your GitHub repository page.
Calls: <Anonymous> ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
In addition: Warning message:
In pb_upload_file(f, repo, tag, name, overwrite, use_timestamps, :
use_timestamps argument is deprecated please set overwrite='use_timestamps' instead.
Execution halted
bba
bba
I have a GITHUB_TOKEN variable in my .Renviron file so not sure why I'm seeing this error.
Awesome - we're so close now!
Hmm, did you generate the GITHUB_TOKEN variable yourself? You might need specify certain permissions when creating it so that it can access the organisations your a part of?
I think I created a token for you and Rich to use ages ago, if you like I could email that to you and you could see if it still works? If so, could you please send me an email so that I have your current address (I can't seem to find your email address in my inbox).
Yes, believe I am using that one that you gave us, and it's worked in the past so I'm not sure what the issue would be.
louis.backstrom@gmail.com still :)
I think the Makefile might be failing to copy the .Renviron file into the Docker container when it tries to push the assets. To verify this, could you please (1) open the command prompt, (2) enter the command sh
, (3) enter the command echo $OS
, and (4) enter the command echo $USERPROFILE
. Next, could you please tell me what the command prompt prints after 3 and 4?
C:\Users\louis>sh
sh-4.4$ echo $OS
Windows_NT
sh-4.4$ echo $USERPROFILE
C:\Users\louis
sh-4.4$
Nothing too odd?
Yeah, that's exactly what my windows computer says
Hmm. Thoughts?
Ok, I've just pushed a commit to add a new command to makefile to test that the Renviron file is being copied across correctly. Could you please pull the latest version from GitHub, open the command prompt, and run the command make test_access
?
Also, could you copy what it says here?
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make test_access
916d75133aa6e444b4393f9db8d4d80eec35278b7a94f42985314062803097be
-rwxr-xr-x 1 root root 226 Jan 29 05:23 /root/.Renviron
bba
bba
Hmm, so it would appear that it's copying the file correctly. I'll try adding some code to try uploading a test file. One moment.
Ok - can you please try running the make test_access
command again?
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make test_access
8aa2a1e0c140f35cbed110e4ab95c70329a86aacd7b240038feab14ac3ea76ce
-rwxr-xr-x 1 root root 226 Jan 29 05:23 /root/.Renviron
&& docker exec bba sh -c "R -e 'f=tempfile();writeLines(\"this is a test2\", f);pb_upload(file=f, repo=\"bird-team/brisbane-bird-atlas\", tag=\"v.0.0.1\", name=\"new-test5.txt\")'" || true
sh: -c: line 0: syntax error near unexpected token `&&'
sh: -c: line 0: `&& docker exec bba sh -c "R -e 'f=tempfile();writeLines(\"this is a test2\", f);pb_upload(file=f, repo=\"bird-team/brisbane-bird-atlas\", tag=\"v.0.0.1\", name=\"new-test5.txt\")'" || true'
make: *** [Makefile:143: test_access] Error 1
Sorry, there was a typo - could you please pull and try again? Also, it's 8:15pm here, so I probably won't look at this again tonight - can we pick this up again tomorrow or next week? It might be useful to try using something like teamviewer so I can connect to your computer directly to try this stuff.
I believe you missed the piggyback::
before pb_upload
, as that was throwing an error, so I added that to the makefile but still the same GITHUB_TOKEN error:
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make test_access
8bf90d4f71c7367c2321c65bc85e358e72f440b78179daef02afd23f6bf1addc
-rwxr-xr-x 1 root root 226 Jan 29 05:23 /root/.Renviron
R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> f=tempfile();writeLines("this is a test2", f);piggyback::pb_upload(file=f, repo="bird-team/brisbane-bird-atlas", tag="v.0.0.1", name="new-test5.txt")
Error in value[[3L]](cond) :
Cannot access release data for repository bird-team/brisbane-bird-atlas. Check that you have set a GITHUB_TOKEN and that at least one release on your GitHub repository page.
Calls: <Anonymous> ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
bba
bba
No rush to fix this up - and yes, we can try something like teamviewer/remote desktop.
Ah - thank you for catching that. Hmm, let's double check that the /root
folder is the right place for the .Renviron file? I've just pushed a commit that prints where R is looking for the .Renviron file when you run make test_access
.
Also, I just stumbled across this resource (https://rstats.wtf/r-startup.html#renviron) which says that:
Both .Renviron and .Rprofile must end with a newline character. If they do not the last line with be ignored without a warning or error.
Could you please confirm if there is a blank line after the GITHUB_TOKEN=XXXXXXX
text in your .Renviron file?
C:\Users\louis\Documents\GitHub\brisbane-bird-atlas>make test_access
11073731b70ad0d57ca1b9f331a6e719bd5bd91d18dd87387726557ec7d5b123
-rwxr-xr-x 1 root root 226 Jan 30 03:25 /root/.Renviron
R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> Sys.getenv("R_PROFILE_USER")
[1] ""
>
>
/root
R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> f=tempfile();writeLines("this is a test2", f);piggyback::pb_upload(file=f, repo="bird-team/brisbane-bird-atlas", tag="v.0.0.1", name="new-test5.txt")
Error in value[[3L]](cond) :
Cannot access release data for repository bird-team/brisbane-bird-atlas. Check that you have set a GITHUB_TOKEN and that at least one release on your GitHub repository page.
Calls: <Anonymous> ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
bba
bba
Right, so I checked the .Renviron file, and it did have a newline character at the end. I moved the GITHUB_TOKEN line to the bottom (I have about half a dozen lines of keys etc. in the file) and ensured there was still a blank line after and still no luck.
Ok - thanks for the checking that. I've pushed another update to the make test_access
command so that it prints the text in the .Renviron file on the Docker image, and the GITHUB_TOKEN variable in R. Can you please verify if they match the stuff in your .Renviron file? Also, please don't post the output here, since it will contain the secret token needed to modify the atlas.
Looks fine to me... GITHUB_TOKEN matches what it is in the .Renviron file.
Going to try deleting the other tokens currently in the .Renviron file (even though I use them all for other projects) to see if that fixes it.
Update: that didn't work either. Same error:
> f=tempfile();writeLines("this is a test2", f);piggyback::pb_upload(file=f, repo="bird-team/brisbane-bird-atlas", tag="v.0.0.1", name="new-test5.txt")
Error in value[[3L]](cond) :
Cannot access release data for repository bird-team/brisbane-bird-atlas. Check that you have set a GITHUB_TOKEN and that at least one release on your GitHub repository page.
Calls: <Anonymous> ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
bba
bba
Ok that's weird. So it seems like the token is being imported correctly. Maybe the Docker container is having trouble accessing the internet? I'll add a test for that in a minute.
Trying to update Atlas assets to the latest version of eBird data (Dec-2020). Docker seems to be playing up:
I have the latest versions of Cygwin, Docker, and Rtools installed and running, everything seems to be in order but when I run
make pull_image
I get the following:Google searching doesn't give me particularly informative solutions to the 127 error.
Thoughts?