Make upload of docker, yum and other artifacts types

[3eka]

What would you like to be added: So, I can backup my docker hosted repository - that works fine. BUT, it seems it is not possible to upload it back to nexus (check table on start page). Same is true for many other types.

Why is this needed: Well, if I can backup (for example) docker hosted repository, I would also like to be able to upload it somewhere else (another nexus instance)

If I missing it (case exists), please explain. I have read documentation (several times), and could not find options. Thanks

[3eka]

Example:

This was done w/o problems:

:; n3dr backup --n3drRepo my-yum --n3drUser predrag_zecevic --n3drPass *** --n3drURL ${REPO1} \
       --directory-prefix /backup/tmp/n3dr
...

:; cd /backup/tmp/n3dr

Failure

:; n3dr upload --n3drRepo my-yum --n3drUser predrag_zecevic --n3drPass *** --n3drURL ${REPO2}
...
INFO[0000]/pz/n3dr/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'maven2'
FATA[0000]/pz/n3dr/cmd/n3dr/upload.go:23 main.glob..func8() no folders with .pom files detected. Please check whether the 'my-yum' directory contains .pom files

:; :; n3dr upload --n3drRepo my-yum --n3drUser predrag_zecevic --n3drPass *** --n3drURL ${REPO2} --artifactType yum
...
INFO[0000]/pz/n3dr/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'yum'
FATA[0000]/pz/n3dr/cmd/n3dr/upload.go:23 main.glob..func8() Upload of 'yum' is not supported

Regards.

[3eka]

And, just noticed (when backing up all repositories, from Nexus OSS 3.34.1-01):

:; n3dr repositories --backup --n3drURL http://host:port/nexus --https false --n3drUser ... --n3drPass ...
...
Nexus repository: 'composer_external', format: 'composer' download not supported
Nexus repository: 'test-raw', format: 'raw' download not supported

I have used repositories instead of repositoriesV2, because second one fails:

it accepts ONLY host:port for --n3drURL, but I have host:port/nexus path.

How to fix that situation (for second Nexus instance it works -- I do not have path after host:port specification)?

Regards.

P.S: It would be nice to have more detailed explanation for each option (--help give just minimum).

[030]

Thank you for addressing this. We will have to fix the following:

• [x] allow all kind of base URLs in conjunction with repositoriesV2, i.e.: host:port/nexus should be allowed as well
• [ ] add more detailed explanation for each option
• [x] improve documentation, e.g. what commands are recommended and which will be deprecated
• [x] increase number of supported formats, e.g.: yum, docker
• [x] test packer, vagrant and run integration test on windows
• [x] rename master to main branch

[030]

@3eka Could you try 6.4.3 in conjunction with --basePathPrefix?

[3eka]

Tested. Seems there is other problem:

:; n3dr repositoriesV2 --backup --n3drURL FQDN:8081 --https false --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr --basePathPrefix=nexus
...
INFO[0000]/pz/n3dr/internal/pkg/connection/connection.go:22 github.com/030/n3dr/internal/pkg/connection.(*Nexus3).Client() adding 'nexus' as a prefix to the basePath
FATA[0000]/pz/n3dr/cmd/n3dr/repositoriesV2.go:42 main.glob..func7() cannot get repository names: 'Get "https://FQDN:8081/nexus/service/rest/v1/repositorySettings": http: server gave HTTP response to HTTPS client'

It tries https (although not specified with --https false), but accepts --basePathPrefix

Regards.

[3eka]

Utility was built from git clone this morning, and returns no version:

:; n3dr --version
n3dr version local

Regards.

[030]

The version is only added by the CI. When I download one of the artifacts the the version is 6.4.3:

n3dr --version
n3dr version refs/tags/6.4.3

[3eka]

I have created 2 nexus nodes, both listening on https, 8443 port:

• OLD (contains 3 repos: yum, docker and raw)
• NEW (created empty repositories, exactly with same configuration as ON OLD one)

Did backup (to empty directory):

:; n3dr repositoriesV2 --backup --n3drURL OLD:8443 --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr

and that resulted in:

:; du -shc /nobackup/tmp/n3dr/my*
24K     /nobackup/tmp/n3dr/my-docker
31K     /nobackup/tmp/n3dr/my-raw
1,6G    /nobackup/tmp/n3dr/my-yum
1,6G    total

NOTES*

• my-docker is NOT OK -- has ~ 600 MB size and was not downloaded completely (e.g.blobs are missing, repository was downloaded)
• my-raw is OK
• my-yum is OK

Upload is still not working (tried mentioned 3 repositories only):

:; n3dr repositoriesV2 --upload --n3drURL NEW:8443 --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr
...
INFO[0000]/pz/n3dr/internal/artifactsv2/upload/upload.go:369 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'NEW:8443' repository: 'my-raw'...
INFO[0000]/pz/n3dr/internal/artifactsv2/upload/upload.go:369 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'NEW:8443' repository: 'my-docker'...
INFO[0000]/pz/n3dr/internal/artifactsv2/upload/upload.go:369 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'NEW:8443' repository: 'my-yum'...
...

Only my-raw was uploaded, my-yum and my-docker, were not (repository on NEW node was created exactly as one on NEW node):

:; du -shc /nobackup/tmp/n3dr/my-yum/
1,6G    /nobackup/tmp/n3dr/my-yum/
1,6G    total

Regards.

[3eka]

The version is only added by the CI. When I download one of the artifacts the the version is 6.4.3:

n3dr --version
n3dr version refs/tags/6.4.3

Hmm,

:; aria2c https://github.com/030/n3dr/archive/refs/tags/6.4.3.tar.gz
:; tar xf n3dr-6.4.3.tar.gz
:; cd n3dr-6.4.3
:; bash scripts/build.sh
:; cmd/n3dr/n3dr --version
n3dr version local

What I am doing wrong?

[030]

@3eka The binary has already been created by the CI:

curl -L https://github.com/030/n3dr/releases/download/6.4.3/n3dr-ubuntu-20.04 -o n3dr && chmod +x n3dr && ./n3dr --version
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   656  100   656    0     0   8000      0 --:--:-- --:--:-- --:--:--  8000
100 22.5M  100 22.5M    0     0  22.7M      0 --:--:-- --:--:-- --:--:-- 32.1M
n3dr version refs/tags/6.4.3

In order to assign a version locally:

GITHUB_TAG=some-version ./scripts/build.sh && ./cmd/n3dr/n3dr --version
GITHUB_TAG: 'some-version' N3DR_DELIVERABLE: 'n3dr'
n3dr version some-version

[030]

@3eka 6.5.0 supports rpm/yum uploads.

[3eka]

Thanks, will test it today. What about problem described here: https://github.com/030/n3dr/issues/238#issuecomment-1009701830 ? Tried, and got same error as described. Regards.

[3eka]

HI @030, something is still NOT OK.

Case 1) backup of everything from OLD nexus instance misses docker repository backup (same problem as described in https://github.com/030/n3dr/issues/238#issuecomment-1009738435). So, complete backup is missing.

Case 2) backup of docker repository (only) works, but not upload (previously an EMPTY docker repository was created)::

:; n3dr backup --n3drRepo my-docker --n3drURL https://OLDFQDN:8443 --n3drUser ... --n3drPass ...  --directory-prefix /nobackup/tmp/n3dr 
...
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/backup.go:34 github.com/030/n3dr/internal/artifacts.TempDownloadDir() Download dir name: '/nobackup/tmp/n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/backup.go:29 main.glob..func1() Processing repository: my-docker
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/backup.go:281 github.com/030/n3dr/internal/artifacts.Nexus3.StoreArtifactsOnDiskChannel() Backing up: 'my-docker'

:; du -shc /nobackup/tmp/n3dr/my-docker
575M    /nobackup/tmp/n3dr/my-docker
575M    total

:; n3dr repositoriesV2 --upload --n3drURL NEWFQDN:8443 --n3drUser predrag_zecevic --n3drPass ${N3DRPASS} --directory-prefix /nobackup/tmp/n3dr
...
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:71 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:159 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/pz/n3dr-6.5.0/internal/artifactsv2/upload/upload.go:372 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'NEWFQDN:8443' repository: 'my-docker'...

Although it throws no error, nothing was uploaded.

Tested also yum repository backup/upload, also no success (but error is shown):

; n3dr backup --n3drRepo my-yum --n3drURL https://OLDFQDN:8443 --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr 1>/dev/null
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:71 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:159 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/backup.go:34 github.com/030/n3dr/internal/artifacts.TempDownloadDir() Download dir name: '/nobackup/tmp/n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/backup.go:29 main.glob..func1() Processing repository: my-yum
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/backup.go:281 github.com/030/n3dr/internal/artifacts.Nexus3.StoreArtifactsOnDiskChannel() Backing up: 'my-yum'

:; du -shc /nobackup/tmp/n3dr/my-yum
1,6G    /nobackup/tmp/n3dr/my-yum
1,6G    total

:; n3dr repositoriesV2 --upload --n3drURL NEWFQDN:8443 --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr 1>/dev/null
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:71 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:159 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/pz/n3dr-6.5.0/internal/artifactsv2/upload/upload.go:372 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'NEWFQDN:8443' repository: 'my-yum'...
FATA[0015]/pz/n3dr-6.5.0/cmd/n3dr/repositoriesV2.go:48 main.glob..func7() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/repodata/dfad0846e198d2b4c08b7462a5657563db3babb3d48345cc3f64ef3299883698-other.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'

This upload option is also not working (same error shown when docker repository upload is attempted):

:; n3dr upload --n3drRepo my-yum --n3drURL https://NEWFQDN:8443 --n3drUser ... --n3drPass ... --directory-prefix /nobackup/tmp/n3dr 1>/dev/null
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:71 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:159 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'maven2'
FATA[0000]/pz/n3dr-6.5.0/cmd/n3dr/upload.go:23 main.glob..func9() lstat my-yum: no such file or directory

:; cd /nobackup/tmp/n3dr
:; n3dr upload --n3drRepo my-yum -t=yum --n3drURL https://NEWFQDN:8443 --n3drUser ... --n3drPass ...
...
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:71 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/pz/n3dr-6.5.0/cmd/n3dr/root.go:159 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/pz/n3dr-6.5.0/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'yum'
FATA[0000]/pz/n3dr-6.5.0/cmd/n3dr/upload.go:23 main.glob..func9() Upload of 'yum' is not supported

Am I missing something?

Regards.

[3eka]

Maybe I can ask for --n3drLogo false switch, to not display logo over whole screen?

[3eka]

Hi @030 ,

how to prevent display of logo?

:; n3dr --help | grep -i logo
      --showLogo                      show N3DR logo or not. Default: true (default true)

Whatever I have tried, it has failed: --showLogo false also does not work.

BTW: I will try version 6.6.1 tomorrow.

[030]

Could you try --showLogo=false? Example: https://github.com/030/n3dr/blob/main/test/integration-tests.sh#L514

[3eka]

Thanks First, that works (--showLogo=false). Does it means that I have to use = between option and its value for everything?

A) backup works:

:; rm -rf /nobackup/tmp/n3dr/*
:; for repo in my-docker my-yum my-raw; do    
     n3dr backup --showLogo=false --n3drRepo=${repo} --anonymous --n3drURL ${NEXUS_O} --directory-prefix /nobackup/tmp/n3dr
  done

:; cd /nobackup/tmp/n3dr
:; du -shc *
3,8G    my-docker
110K    my-raw
5,8G    my-yum

B) load of raw repository work, other not (${NEXUS_N} has configured hosted docker, raw and yum EMPTY repositories. Nexus is available directly, e.g. no /blahblah in its URL):

:; n3dr upload --showLogo=false --artifactType=docker --n3drURL=${NEXUS_N} --n3drUser=predrag_zecevic --n3drPass=***  --n3drRepo my-docker
INFO[0000]/n3dr/cmd/n3dr/root.go:76 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr' 
INFO[0000]/n3dr/cmd/n3dr/root.go:166 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml' 
INFO[0000]/n3dr/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'docker'                           
FATA[0000]/n3dr/cmd/n3dr/upload.go:23 main.glob..func9() Upload of 'docker' is not supported

:; n3dr upload --showLogo=false --artifactType=yum --n3drURL=${NEXUS_N}  --n3drUser=predrag_zecevic --n3drPass=*** --n3drRepo my-yum
INFO[0000]/n3dr/cmd/n3dr/root.go:76 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr' 
INFO[0000]/n3dr/cmd/n3dr/root.go:166 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml' 
INFO[0000]/n3dr/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'yum'                              
FATA[0000]/n3dr/cmd/n3dr/upload.go:23 main.glob..func9() Upload of 'yum' is not supported

Regards.

P.S: I used this to compile tool:

N3DR_V=6.6.1
    wget --continue https://github.com/030/n3dr/archive/refs/tags/${N3DR_V}.tar.gz
    tar xf ${N3DR_V}.tar.gz && cd n3dr-${N3DR_V}

    GITHUB_TAG=${N3DR_V} bash scripts/build.sh

Still, it does not show version:

:; go version
go version go1.17.9 illumos/amd64

 :; n3dr --version
n3dr version local

[3eka]

HI! Any news on this? I have compiled n3dr few days ago, and I am still unable to upload yum and docker repositories, taken from one nexus instance to this new one. Both instances OLD and NEW are running same Nexus s/w (3.39.0-01) and both have configured repositories. I am able to do backup (example for yum repository -- docker one shows same problem):

:; n3dr backup --n3drRepo=yum --anonymous --n3drURL=https://old:8443 --directory-prefix=/nobackup/tmp/n3dr --showLogo=false
:; cd /nobackup/tmp/n3dr
:; du -sh yum
7,0G    yum/

Tying to upload:

:; n3dr upload --showLogo=false --n3drURL=https://new:8443 --n3drUser=... --n3drPass=... --artifactType=yum --n3drRepo=yum --directory-prefix=/nobackup/tmp/n3dr
INFO[0000]/n3dr/cmd/n3dr/root.go:76 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/n3dr/cmd/n3dr/root.go:166 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/n3dr/internal/artifacts/upload.go:364 github.com/030/n3dr/internal/artifacts.Nexus3.Upload() Uploading 'yum'
FATA[0000]/n3dr/cmd/n3dr/upload.go:23 main.glob..func9() Upload of 'yum' is not supported

What is proper syntax to upload? Am I doing something wrong? Regards.

P.S: Docs page does not reflect output from n3dr --help command.

[030]

Did you try repositoriesV2 as well?

[3eka]

Yes, not working either:

/nobackup/tmp/n3dr :; du -sh *
5,7G    my-docker
110K    my-raw
7,0G    my-yum

and this is "import":

:; cd /nobackup/tmp/n3dr
:; n3dr repositoriesV2 --upload --showLogo=false --n3drURL=nexus-node.mydomain.com:8443 --n3drUser=predrag_zecevic --n3drPass=*** --directory-prefix=/nobackup/tmp/n3dr
INFO[0000]/n3dr/cmd/n3dr/root.go:76 main.n3drHiddenHome() n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000]/n3dr/cmd/n3dr/root.go:166 main.parseConfig() Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
INFO[0000]/n3dr/internal/artifactsv2/upload/upload.go:62 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).reposOnDisk() found the following localDiskRepos: '[my-yum my-docker my-raw]'
INFO[0000]/n3dr/internal/artifactsv2/upload/upload.go:320 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'nexus-node.mydomain.com:8443' repository: 'my-raw'...
INFO[0000]/n3dr/internal/artifactsv2/upload/upload.go:320 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'nexus-node.mydomain.com:8443' repository: 'my-yum'...
INFO[0000]/n3dr/internal/artifactsv2/upload/upload.go:320 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() Uploading files to Nexus: 'nexus-node.mydomain.com:8443' repository: 'my-docker'...
INFO[0000]/n3dr/internal/artifactsv2/upload/upload.go:80 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).repoFormatLocalDiskRepo() format of repo: 'my-raw' is: 'raw'
INFO[0001]/n3dr/internal/artifactsv2/upload/upload.go:80 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).repoFormatLocalDiskRepo() format of repo: 'my-docker' is: 'docker'
WARN[0001]/n3dr/internal/artifactsv2/upload/upload.go:326 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).Upload.func1() docker and snapshot repositories skipped
INFO[0001]/n3dr/internal/artifactsv2/upload/upload.go:80 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).repoFormatLocalDiskRepo() format of repo: 'my-yum' is: 'yum'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/SRPMS/repodata/repomd.xml', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/SRPMS/maven-3.8.4-1.src.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/repodata/ff3cdbcf25f89c6c2a526d2a40d62919ff17274f307e744b8124ae61b5fb1824-filelists.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/repodata/f1e0ab9b1546d344db515e18fc56ebe927874cb99f62673bbc91841838b60322-primary.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/8/x86_64/RPMS/sudosh2-1.0.7-1.2e.x86_64.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/8/x86_64/RPMS/cronolog-debuginfo-1.7.2-1.x86_64.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/cronolog-1.7.2-1.x86_64.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/repodata/f511f663d3b08d7bcbc64f37d163d19cea154c574804d8bb0fe8fc04b2556b4d-other.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/cronolog-debuginfo-1.7.2-1.x86_64.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/SRPMS/repodata/d7bb659c1322c3e8fb37d02fb6e984e52d8e27355db6b5c11e4044e220aa8ed9-other.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/SRPMS/repodata/8dc2749e3926c726bc0a8cbdf219e0a96ae927dbf15c90b04ea6fbe15c99d430-primary.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/8/x86_64/RPMS/repodata/ea45fe64ff71036028d9e2a92f5c3d0e4cdcd73768fdfed900fac1e89675253d-primary.xml.gz', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/repodata/repomd.xml', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/7/x86_64/SRPMS/repodata/repomd.xml', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
ERRO[0001]/n3dr/internal/artifactsv2/upload/upload.go:283 github.com/030/n3dr/internal/artifactsv2/upload.(*Nexus3).ReadLocalDirAndUploadArtifacts.func1.1() cannot upload component: '/nobackup/tmp/n3dr/my-yum/centos/6/x86_64/RPMS/php-devel-7.3.24-1.ssl1.1.1h.x86_64.rpm', error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec (status 400): {}'
...

it recognizes type of data, but load fails. When I open https://nexus-node.mydomain.com:8443 I am able to log-in, browse, etc

So, what exactly means error: 'response status code does not match any response statuses defined for this endpoint in the swagger spec error?

Regards

[3eka]

After switching to

:; n3dr --version
n3dr version 6.7.3         # logo file changed
:; go version
go version go1.19.1 illumos/amd64

Backup started to fail (nexus node listens to https port 8443):

:; n3dr repositoriesV2 --backup --directory-prefix /nobackup/tmp/n3dr/nexus --n3drUser=predrag_zecevic --n3drPass=*** --n3drURL=nexus.my-domain.com:8443
INFO[0000] n3drHomeDir: '/export/home/predrag_zecevic/.n3dr' 
INFO[0000] Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml' 
INFO[0000] false                                        
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'my-yum', 'hosted', yum           
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'nuget-group', 'group', nuget     
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'maven-snapshots', 'hosted', maven2 
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'maven-central', 'proxy', maven2  
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'nuget.org-proxy', 'proxy', nuget 
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'maven-releases', 'hosted', maven2 
INFO[0000]/n3dr/internal/artifactsv2/download.go:164 github.com/030/n3dr/internal/artifactsv2.(*Nexus3).Backup() backing up 'my-docker', 'hosted', docker     
FATA[0000]/n3dr/cmd/n3dr/repositoriesV2.go:42 main.glob..func7() parse ":0/v2/_catalog": missing protocol scheme

I guess it is stuck on my-docker repository (from some reason... was working before -- does go version has something to do with?), because we see under output directory '/nobackup/tmp/n3dr/nexus':

my-yum
maven-central
maven-releases
maven-snapshots
nuget.org-proxy

With --debug switch, it outputs only this line (real password is replaced with ***; I connect with that account w/o problems to web interface):

[0000]/export/home/predrag_zecevic/go/pkg/mod/github.com/030/p2iwd@v0.2.0/internal/pkg/http/http.go:20 github.com/030/p2iwd/internal/pkg/http.(*Auth).RequestAndResponse() Auth: '&{Accept  GET *** :0/v2/_catalog predrag_zecevic}'

I really like idea behind this project, but I am kind stuck (wanted to do nexus backup/restore with n3dr).

Best regards.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[3eka]

I will add soon new tests result...

[3eka]

Testing (latest?) version shows same symptoms as already described here: https://github.com/030/n3dr/issues/238#issuecomment-1262076140 (download of docker repo fails/terminates program).

:; n3dr --version
n3dr version 7.0.2

[030]

@3eka I get the same issue: FATA[0000]/home/runner/work/n3dr/n3dr/cmd/n3dr/repositoriesV2.go:89 main.glob..func5() parse ":0/v2/_catalog": missing protocol scheme when I have omitted the --dockerPort 9001 and --dockerHost http://localhost. Could you check whether you have set these parameters? I will update the code to ensure that an error will be returned if the docker host and or port have not been set.

[3eka]

@3eka I get the same issue: FATA[0000]/home/runner/work/n3dr/n3dr/cmd/n3dr/repositoriesV2.go:89 main.glob..func5() parse ":0/v2/_catalog": missing protocol scheme when I have omitted the --dockerPort 9001 and --dockerHost http://localhost. Could you check whether you have set these parameters? I will update the code to ensure that an error will be returned if the docker host and or port have not been set.

Hi @030 ;

Yes, it seems that adding docker options have had a progress here...

Question regarding upload: shall I use also docker switches for upload/restore?

Thanks for fixing this. Regards.

[3eka]

Also, usage of n3dr repositoriesV2 --backup options is inconsistent: --n3drURL=mynexus.mydomain.com:PORT (no protocol) but --dockerHost=PROTOCOL://mynexus.mydomain.com --dockerPort=PORT (specifies protocol) NOTE, there is also --https switch available

[3eka]

And:

      --logLevel string               change the log level (default: info, options: trace, debug, info, warn, error or none) (default "info")
      --showLogo                      show N3DR logo or not. Default: true (default true)

Default value shown twice

[3eka]

Then: I have specified:

--logFile=/nobackup/n3dr/n3dr_backup.log
--logLevel=info

But still some output is written to stdout, for example:

INFO[0000] n3drHomeDir: '/export/home/predrag_zecevic/.n3dr'
INFO[0000] Using config file: '/export/home/predrag_zecevic/.n3dr/config.yml'
time="2023-01-04T16:09:50+01:00" level=info msg="backing up 'my-yum', 'hosted', yum" func="github.com/030/n3dr/internal/app/n3dr/artifactsv2.(*Nexus3).Backup" file="/n3dr/internal/app/n3dr/artifactsv2/download.go:233"
...
##%%%%%%%%%%#%############################################################

Which is kind annoying

[3eka]

@3eka I get the same issue: FATA[0000]/home/runner/work/n3dr/n3dr/cmd/n3dr/repositoriesV2.go:89 main.glob..func5() parse ":0/v2/_catalog": missing protocol scheme when I have omitted the --dockerPort 9001 and --dockerHost http://localhost. Could you check whether you have set these parameters? I will update the code to ensure that an error will be returned if the docker host and or port have not been set.

Hi @030 ,

I have added docker options, but still my docker repository is NOT backed-up, e.g. my-docker directory is not there.

Any other suggestions? Regards.

[030]

@3eka do you see a p2iwd folder?

[3eka]

Nope (if you meant under --directory-prefix directory?).

[3eka]

Arghh, I just figured out that I need that utility for backing-up docker images too... Right?

Anyway, will download it, build and test... if that is correct thinking

[3eka]

NOTE: added p2iwd to the PATH and issued backup command again. No luck

[030]

@3eka no need to add p2iwd to the path. If --directory-prefix is used then several folders should be created: one for every repository and there should be a p2iwd folder that includes the docker repositories.

[kcathey]

I'm having this same issue with docker images not being downloaded. There is no my-docker or p2iwd directory when doing a backup of the entire nexus.

When i do a backup of a specific docker repository I get a my-docker folder with just the manifests, no images.

[030]

@kcathey and @3eka Could you try the docker-quickstart and compare it with the parameters you have used?

[kcathey]

So I discovered that Nexus doesn't expose the images that are proxied, only those that have be explicitly pushed. I was trying to do an export of my proxied images, to avoid explicitly pushing them. When there are only proxied images in Nexus the /v2/_catalog endpoint returns an empty list, as soon as an image is pushed it returns a list with the pushed image.

So for my part this is no longer a problem and in either case it isn't yours, but seems to be on the Nexus end.

[3eka]

Hi @kcathey,

can you elaborate "as soon as an image is pushed" task? We are uploading docker images to the nexus, after those are built on jenkins. That is not pushing, right?

Thanks for checking this. Regards.

P.S: Seems that we need to change way of uploading images (with pushing)

[kcathey]

I do a docker push <nexus tag> to upload to nexus. As far as I know that's the only way to upload a docker image.