Store attachments in external repository

[bs-jokri]

From @bs-jokri on December 12, 2016 10:3

Instead of manually uploading the source code attachments into sw360, one wants to have the option to use an external repository for data-keeping. For this one might have to add an UI element, which allows on-demand adding of credentials.

Copied from original issue: bsinno/sw360#364

[bs-jokri]

From @maxhbr on December 12, 2016 15:40

e.g.: https://www.sonatype.com/products-sonatype

[bs-jokri]

This should work for

• Sonatype Nexus Repo and
• Artifactory

[bs-jokri]

https://www.jfrog.com/confluence/display/RTF/Installing+on+Linux+Solaris+or+Mac+OS

https://books.sonatype.com/nexus-book/reference/installing.html

[bs-jokri]

UC1 link release to artifact in repo

1. create release 2.a. click add attachments 2.b. select artifact in external repo 2.b.1 give warning that artifact must be accessible by public 2.c. select attachment type 3 save

UC2. Use externally linked artifact in sw360 functionality (generate source code bundle, disc doc)

1. start function
2. check all attachments if repo requries password 2.a if pw required request from user (it could be that different repos can be )
3. function reads attachment 2a if attachment is in external repo and requires password request password from user 2b buffer password

[bs-jokri]

From @maxhbr on January 25, 2017 15:50

Idea: add metadata for each repository and save url, and isPasswordRequired

[bs-jokri]

From @maxhbr on January 30, 2017 12:38

Nexus Sonatype:

rest API: https://oss.sonatype.org/nexus-restlet1x-plugin/default/docs/index.html

See:

To test for existence:

• curl http://localhost:8081/nexus/service/local/status
• curl -u admin:admin123 http://localhost:8081/service/metrics/ping

Docker: https://hub.docker.com/r/sonatype/nexus/ Nexus3 pdf: https://books.sonatype.com/nexus-book/pdf3/nxbook-pdf.pdf

repositories

raw (at localhost:8081/repository/raw/)

url looks like localhost:8081/repository/raw/antlr4-4.6.tar.gz and basicAuth is requested.

[bs-jokri]

From @maxhbr on January 30, 2017 14:11

Artifactory:

AQL: https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language Advanced REST: https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API

test setup

docker

$ docker pull docker.bintray.io/jfrog/artifactory-oss:latest
$ docker run --name artifactory -d -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss:latest

curl api

upload

$ curl -uadmin:AP4aLhiuB6DKcDvv4no1qW3AVtt -T antlr4-4.6.tar.gz "http://localhost:8082/artifactory/generic-local/antlr4-4.6.tar.gz" 
{
  "repo" : "generic-local",
  "path" : "/antlr4-4.6.tar.gz",
  "created" : "2017-02-07T12:20:11.292Z",
  "createdBy" : "admin",
  "downloadUri" : "http://localhost:8082/artifactory/generic-local/antlr4-4.6.tar.gz",
  "mimeType" : "application/x-gzip",
  "size" : "3781141",
  "checksums" : {
    "sha1" : "c1a5f704ca598c421e9fa72f1db0499c1a17ad08",
    "md5" : "7b6d4f97a1b0f886c7656b70fa665fa9"
  },
  "originalChecksums" : {
  },
  "uri" : "http://localhost:8082/artifactory/generic-local/antlr4-4.6.tar.gz"
}

download

$ curl -uadmin:AP4aLhiuB6DKcDvv4no1qW3AVtt -O "http://localhost:8082/artifactory/generic-local/antlr4-4.6.tar.gz"                   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3692k  100 3692k    0     0  53.2M      0 --:--:-- --:--:-- --:--:-- 53.8M

[bs-jokri]

From @maxhbr on February 7, 2017 10:8

Generic aproach

allow attachments do be stored in external repository and prevent syncing to SW360-couchdb. This implies that an Attachment has essentially three states

• NOT_REMOTE (only copy is local, default)
• LAZY_NOT_REMOTE (remote URL is given, will be synced to couchdb on the first usage)
• ONLY_REMOTE (will never be stored in local DB)

plus

• is now NOT_REMOTE but was LAZY_NOT_REMOTE (might be updatable)

Questions

• compute hash of remote attachments, i.e. download them once and pipe them through.
  • where is the hash used?
  • use this to force the LAZY_... ones?

(generic-) Problems

• [ ] auth?
• [ ] pasted URLs with expiring token
• [ ] PROXY??
• [ ] Zertifikate?
• [ ] no upload!

[bs-jokri]

From @maxhbr on June 21, 2017 8:15

upstream issue is https://github.com/sw360/sw360portal/issues/465

[bs-jokri]

From @maxhbr on July 7, 2017 12:0

Notes from discussion

• specific connectors for artifact repositories (e.g. Nexus, Artifactory with their repr. versions)
• only preconfigured artifact repositories are allowed and selectable in the UI
  • add a possibility to configure remotes
• offer a form to reference artifacts (instead of arbitrary URLs)
• use a technical user to connect to the repositories
  • this user should have only the minimal permissions, which are also assigned to all other logged-in users
• add a plug-in system to deploy implementations for different repository types and versions
  • e.g. via thrift service discovery, via new deployment

    Next step

  • enable upload via SW360?

[bs-jokri]

Text from duplicate issue https://github.com/sw360/sw360portal/issues/465

If one plans to sync data from an artifact repository server like Sonatype Nexus or Artifactory it would be redundant to save the source/binary attachments also in SW360. It might even be not possible due to the size of the attachments.

This could be solved by introducing remote attachments, which might be verified by their hash (which can be obtained from the repositories vie REST, or it could be provided for the upload).

[bs-jokri]

PR is still not upstream. Needs to be discussed what to do with that.

[TristanFAURE]

Hi any updates of this issue ? Really like the idea to link to nexus or artifactory

[maxhbr]

Sadly this is a realy hard issue, and we have encountered some unexpected problems, e.g.

• how does someone add some artifact from the repository
  • by ID => we have to know the internals of the specific repository and the api
  • by link => how to prevent users from adding links to the overwiew instead of the real artifact or artifacts which are not publically visible
• how to handle permissions
• how to verify that the remote artifact has not changed
  • ask api for a hash => would again be specific to only one solution
  • download imediately after adding => all the advantages are gone
• should one store artifacts in our local database once they were downloaded at some point?
• we have some fragile processes which work with multiple attachments in a row and are not able to handle "missing" artifacts, e.g. generate an usefull error message
• what if someone changes / deletes an attachment on nexus / artifactory

In the sum that results in something which is very hard to accomplish.