Gitlab job hanging when pushing sif with oras to container registry #989

Open rachelwaldon opened 1 year ago

What happened in your environment?

Inside Gitlab CI job trying to push the sif with oras registry results in a 1 hour hanging job after the push command

build-sif:
  stage: build
  timeout: 2h
  script:
    - apptainer version
    - pwd
    - whoami
    - echo $CI_REGISTRY_PASSWORD | apptainer remote login --username $CI_REGISTRY_USER --password-stdin docker://$CI_REGISTRY
    - apptainer build sst.sif sst.def
    - strace -f apptainer -d push sst.sif oras://${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:latest
  tags: [ ubuntu_vm ]

Here is the log strace2.txt

What did you expect to happen?

I expected the push command to be successful resulting in pushed image to Gitlab CI registry

How can we reproduce it?

apptainer version 1.1.7

Running a GitLab job on a virtual machine Ubuntu 22.04 to build an apptainer sif image.

What is the version of your ORAS CLI?

Version: 1.0.0 Go version: go1.20.2 Git commit: b58e7b910ca556973d111e9bd734a71baef03db2 Git tree state: clean

What is your OS environment?

Ubuntu 22.04

Are you willing to submit PRs to fix it?

[ ] Yes, I am willing to fix it.

@rachelwaldon This repository only contains oras command line application. The issue you met is caused by Apptainer, which is built upon v1 release of oras-go SDK, can you help move this issue to oras-go repository?

Also I am not sure if oras is available in your CI/CD pipeline. If it's there, pushing the sif can be easily done via

oras push ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:latest sst.sif

Thanks for the quick reply and for the help. I was able to push with your command. Thanks much!

I couldn't tell if the error was from apptainer or oras? I filed a bug report under apptainer too. I can't do an official transfer because I don't have write access to the repo? I can close this and open it in the oras-go repo? Thanks.

@srkohn

Since sif files are usually large, oras CLI with a progress bar and a speed indicator may help.

enabling debug output might help too

enabling debug output might help too

Current debug output shows HTTP header but not body, user might still see no output in debug when pushing pends for uploading the big HTTP body.

Maybe we should show some stats, like time used and bytes read and pushed from the body and introduce periodically update to display it (considering this happens in CI, not a interactive terminal)

oras-project / oras