hit a panic: runtime error

[ryao-mdanderson]

Hello:

I created a docker image rstudio-4.0.4-20211017, I wanted to converted it to a singularity file. I used the following method and it worked perfect for me in past, but today it failed with the error message:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity rstudio-4.0.4-20211017

Image Format: squashfs Docker Image: rstudio-4.0.4-20211017

Inspected Size: 2883 MB

(1/10) Creating a build sandbox... (2/10) Exporting filesystem... (3/10) Creating labels... (4/10) Adding run script... (5/10) Setting ENV variables... (6/10) Adding mount points... (7/10) Fixing permissions... (8/10) Stopping and removing the container... (9/10) Building squashfs container... INFO: Starting build... INFO: Creating SIF file... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x5635685b337a]

goroutine 1 [running]: github.com/sylabs/sif/v2/pkg/sif.CreateContainerAtPath(0x7ffccd453e35, 0x37, 0xc000586b80, 0x1, 0x1, 0xc0004fef00, 0x2, 0x4) github.com/sylabs/sif/v2@v2.0.0/pkg/sif/create.go:269 +0x17a github.com/sylabs/singularity/internal/pkg/build/assemblers.createSIF(0x7ffccd453e35, 0x37, 0xc0003b0000, 0xc00016c690, 0x2d, 0x0, 0x563568cd707e, 0x5, 0x0, 0x0) github.com/sylabs/singularity@v0.0.0/internal/pkg/build/assemblers/sif.go:121 +0x809 github.com/sylabs/singularity/internal/pkg/build/assemblers.(SIFAssembler).Assemble(0xc000388de0, 0xc0003b0000, 0x7ffccd453e35, 0x37, 0x0, 0x0) github.com/sylabs/singularity@v0.0.0/internal/pkg/build/assemblers/sif.go:212 +0x600 github.com/sylabs/singularity/internal/pkg/build.(stage).Assemble(...) github.com/sylabs/singularity@v0.0.0/internal/pkg/build/stage.go:42 github.com/sylabs/singularity/internal/pkg/build.(Build).Full(0xc0001aa180, 0x5635694e2980, 0xc000042058, 0x0, 0x0) github.com/sylabs/singularity@v0.0.0/internal/pkg/build/build.go:466 +0xb85 github.com/sylabs/singularity/cmd/internal/cli.runBuildLocal(0x5635694e2980, 0xc000042058, 0x563569d6e8a0, 0x7ffccd453e35, 0x37, 0x7ffccd453e6d, 0x39) github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/build_linux.go:348 +0x840 github.com/sylabs/singularity/cmd/internal/cli.runBuild(0x563569d6e8a0, 0xc000264140, 0x2, 0x2) github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/build_linux.go:134 +0x34a github.com/spf13/cobra.(Command).execute(0x563569d6e8a0, 0xc00003c0e0, 0x2, 0x2, 0x563569d6e8a0, 0xc00003c0e0) github.com/spf13/cobra@v1.2.1/command.go:860 +0x2ac github.com/spf13/cobra.(Command).ExecuteC(0x563569d792a0, 0x563569495578, 0xc0000a3860, 0xc00031dfb0) github.com/spf13/cobra@v1.2.1/command.go:974 +0x35c github.com/spf13/cobra.(Command).Execute(...) github.com/spf13/cobra@v1.2.1/command.go:902 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.2.1/command.go:895 github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity() github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:430 +0x20e main.main() github.com/sylabs/singularity@v0.0.0/cmd/singularity/cli.go:19 +0x52

Today, before seeing the above error message, I tried the above command and hit an error: FATAL: While performing build: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /tmp/rstudio-4.0.4-20211017-2021-10-18-ae6bf9b2ebd6.sif: no space left on device

So, I used the command "docker system prune -a" to remove exited containers and unused images. I also restarted docker.

Thank you very much for your suggestion!

Regards, Rong

[vsoch]

I would try the previously working version of docker2singularity. If it fails it could be a space issue. Better yet you can pull directly from a Docker registry with singularity pull docker://image-uri

[ryao-mdanderson]

@vsoch:

Thank you for your reply. I will try your first suggestion. How can I get the previous working version of docker2singularity?

The last time I used docker2singularity was on 2021.09.14, I used the command: docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity rstudio-4.0.4-20210914 (where rstudio-4.0.4-20210914 is a docker image that I modified and created on 2021.09.14).

"docker images" command on the systems now shows: REPOSITORY TAG IMAGE ID CREATED SIZE rstudio-4.0.4-20211017 latest 3a6abac08567 9 hours ago 2.88 GB quay.io/singularity/docker2singularity latest 1264dd7994b2 6 weeks ago 416 MB ......

Specifically, what should be the change in my above command? Thanks!

[vsoch]

Choose an older tag.

[ryao-mdanderson]

@vsoch:

May you please walk me through how to locate an older tag in my case? I am not experienced on this. Thank you!

[vsoch]

Sure, the tags are on the image registry: https://quay.io/repository/singularity/docker2singularity?tab=tags. Try to remember the date when it worked, and choose a tag around then. Try a few. The tag goes after the image name, e.g., quay.io/singularity/docker2singularity:<tag>

[ryao-mdanderson]

@vsoch Thanks again, this helps.

The latest version and the newer version v3.8.2 and v3.8.3 are about 2 months ago. Recall my latest successful work was last month 2021.09.14, I guess I have already used one of these version. I pulled an older version 3.8.1 and tried the above command again, this time, hit the error: While performing build: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /tmp/rstudio-4.0.4-20211017-2021-10-18-2142069924bd.sif: no space left on device

This is the same error I saw before I did "docker system prune -a". I am sorry to bother you, how do I get around to this? "df -h" command shows the following, I am puzzled where could be the issue?

df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 1.8M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/VG0-rootlv 10G 8.7G 1.4G 87% / /dev/mapper/VG0-usrlv 10G 7.8G 2.3G 78% /usr /dev/sda1 477M 180M 268M 41% /boot /dev/mapper/VG0-varlv 60G 3.4G 57G 6% /var /dev/mapper/VG0-tmplv 10G 34M 10G 1% /tmp /dev/mapper/VG0-optlv 10G 5.6G 4.5G 56% /opt /dev/mapper/VG0-homelv 70G 41G 30G 58% /home

[vsoch]

You don’t have enough room on your filesystem. Make more room or try somewhere else. I also already suggested pulling directly from a Docker registry.

[ryao-mdanderson]

@vsoch: I appreciate your help.

Correct me please, from the df -h command, /var has used only 6%, it total has 60G, 57G available. In linux system the images are stored in /var/lib/docker, I am puzzled why /var does not have space for a 2.8G rstudio container (docker image: rstudio-4.0.4-20211017, size 2883M)?

"pulling directly from a Docker registry" -- what does this mean?

[vsoch]

Push the container to a Docker registry and pull down with Singularity, as I showed you earlier. If you need further help please join a singularity slack and ask there.

[ryao-mdanderson]

@vsoch

Sure. I will try this. Thanks again for your helps.

[ryao-mdanderson]

@vsoch: I am sorry to bother you for help.

On my Redhat 7.9 system, I have an image rstudio-4.0.4-20211017 REPOSITORY TAG IMAGE ID CREATED SIZE rstudio-4.0.4-20211017 latest 3a6abac08567 15 hours ago 2.88 GB

I have an account on docker.io, I have a repository hpcteam/rstudio-4.0.4 and want to hold this image.

What's the correct command to allow rstudio-4.0.4-20211017 to be pushed to docker.io, the repository?

Thank you.

[vsoch]

docker tag <your image name> hpcteam/rstudio-4.0.4 
docker push hpcteam/rstudio-4.0.4 

And you probably will need to login, docker login.

[ryao-mdanderson]

@vsoch: I really appreciate for your quick response.

After I did a tag $ docker tag rstudio-4.0.4-20211017 hpcteam/rstudio-4.0.4:20211017

I am able to find the new tagged image [ryao@r1drpswdev3 ~]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE hpcteam/rstudio-4.0.4 20211017 3a6abac08567 15 hours ago 2.88 GB rstudio-4.0.4-20211017 latest 3a6abac08567 15 hours ago 2.88 GB

Then I tried to push $ docker push hpcteam/rstudio-4.0.4 The push refers to a repository [registry.access.redhat.com/hpcteam/rstudio-4.0.4] 5fee665b2ff6: Preparing 2743b040803e: Preparing 2c991c434766: Preparing dac39b3acc9f: Preparing b696421832a4: Preparing 05f3b67ed530: Waiting ec1817c93e7c: Waiting 9e97312b63ff: Waiting e1c75a5e0bfa: Waiting error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "\n\n\n

Access Denied

\n \nYou don't have permission to access \"http://registry.access.redhat.com/v2/hpcteam/rstudio-4.0.4/blobs/uploads/\" on this server.

\nReference #18.85911160.1634587117.3812c44d\n\n\n"

I noticed that I should I login, so I login docker.io: $ docker login docker.io Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: hpcteam Password: Login Succeeded $ docker push hpcteam/rstudio-4.0.4 The push refers to a repository [registry.access.redhat.com/hpcteam/rstudio-4.0.4] 5fee665b2ff6: Preparing 2743b040803e: Preparing 2c991c434766: Preparing dac39b3acc9f: Preparing b696421832a4: Preparing 05f3b67ed530: Waiting ec1817c93e7c: Waiting 9e97312b63ff: Waiting e1c75a5e0bfa: Waiting error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "\n\n\n

Access Denied

\n \nYou don't have permission to access \"http://registry.access.redhat.com/v2/hpcteam/rstudio-4.0.4/blobs/uploads/\" on this server.

\nReference #18.7c911160.1634587314.2624f38a\n\n\n"

Did I ignore anything in this case? I believe the permission repository hpcteatm/rstudio-4.0.4 in https://hub.docker.com/repositories is public.

[vsoch]

Looks like you don't have permission to push to your org.

[ryao-mdanderson]

@vsoch: Thanks! I notice the first line of the push output: The push refers to a repository [registry.access.redhat.com/hpcteam/rstudio-4.0.4] is this supposed to be docker.io/hpcteam/rstudio-4.0.4, do I misconfigure anything on the server side?

[vsoch]

That means your docker install defaults to that registry (not typical so I didn't include it). I'd try tagging the image with docker.io and then re-doing the push.

[ryao-mdanderson]

@vsoch 👍 awesome.

Followed your suggestion I tag the image with docker.io and redo the push, it works. Thank you very much for your help. I guess docker was installed on the system by an rpm provided by redhat, thus it access registry.access.redhat.com by default.

[vsoch]

Ha! That's very funny, but in retrospect makes sense. Glad you got it working. Now try pulling down to Singularity:

$ singularity pull docker://docker.io/hpcteam/rstudio-4.0.4

or this will probably work as singularity defaults to Docker Hub

$ singularity pull docker://hpcteam/rstudio-4.0.4

[ryao-mdanderson]

@vsoch 👍

I used singularity pull command singularity pull docker://docker.io/hpcteam/rstudio-4.0.4:20211017 successfully pulled an image. I will test that shortly. Again, you have saved my day! I really appreciate your patience with me.

[vsoch]

Yep sure thing! Glad it worked. For the future, when you need this kind of back and forth support I recommend a Singularity slack - there are two now :)

[ryao-mdanderson]

@vsoch: Right. I will do. This time I was originally stuck on docker2singularity convert container, glad you point to a right direction. Have a good evening.

[vsoch]

Thanks! Closing the issue since we got you settled. Have a good one!