Upgrade V2 to SW360 11

[oheger-bosch]

This PR updates the experimental v2 version of sw360chores to be compatible with a recent SW360 11 version. It adds missing frontend images, so that a full local SW360 instance can be setup and started via docker-compose. The resulting docker images use the following versions:

• Liferay 7.3.3-ga4
• Thrift 0.13
• CouchDB 2.1.2
• SW360 > 11.0.0-M1 (Initially, I wanted to use the tagged revision, but it turned out that it still had a problem with the CouchDB-Lucene integration; so I used the commit which fixed this problem.)

While I was able to start a local and optimized (due to the minimized deployment artifacts) version of SW360 based on these images successfully, there is probably still much work needed to make this fool-proof in all possible environments. I would therefore like to ask the community to play around with this approach and give feedback what is missing and what does not work as expected.

[neubs-bsi]

@mcjaeger can this be merged?

[mcjaeger]

Hm, for it is not build, what happens:

1. sudo docker-compose build
2. [INFO] BUILD SUCCESSfor all sw360 artefacts in intermediate container
3. tomcat image OK
4. couchdb config and entry point script OK
5. Successfully tagged sw360/backend:latest
6. Building sw360backend-rest, oberservation:

   Step 8/35 : RUN mkdir /maven
   ---> Running in 6352e18dbb83
   Removing intermediate container 6352e18dbb83
   ---> 2a67d63bd52f
   Step 9/35 : RUN cd sw360 &&     mvn -Dmaven.repo.local=/maven $(eval echo "${MVN_FLAGS}" ) install -P deploy -Dbase.deploy.dir=/sw360chores -DskipTests     -pl 'build-configuration,libraries,libraries/lib-datahandler,libraries/commonIO'
   ---> Running in f1fe2f1ab1ec
   basename: missing operand
   Try 'basename --help' for more information.
   basename: missing operand
   Try 'basename --help' for more information.
   basename: missing operand
   Try 'basename --help' for more information.
   basename: missing operand
   Try 'basename --help' for more information.
   [INFO] Scanning for projects...
   Downloading from central: https://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/4.2.1/maven-bundle-plugin-4.2.1.pom
   Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/1.5.8.RELEASE/spring-boot-starter-parent-1.5.8.RELEASE.pom
   [ERROR] [ERROR] Some problems were encountered while processing the POMs:
   [ERROR] Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:4.2.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:4.2.1 @ 
   [ERROR] Non-resolvable import POM: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/1.5.8.RELEASE/spring-boot-starter-parent-1.5.8.RELEASE.pom @ org.eclipse.sw360:authorization-server:${revision}, /sw360/rest/authorization-server/pom.xml, line 158, column 25
   [ERROR] 'dependencies.dependency.version' for org.springframework.security.oauth:spring-security-oauth2:jar is missing. @ org.eclipse.sw360:authorization-server:${revision}, /sw360/rest/authorization-server/pom.xml, line 70, column 21
   [ERROR] 'dependencies.dependency.version' for org.springframework.security:spring-security-jwt:jar is missing. @ org.eclipse.sw360:authorization-server:${revision}, /sw360/rest/authorization-server/pom.xml, line 81, column 21
   [ERROR] 'dependencies.dependency.version' for org.springframework.security:spring-security-test:jar is missing. @ org.eclipse.sw360:authorization-server:${revision}, /sw360/rest/authorization-server/pom.xml, line 104, column 21
   [ERROR] Non-resolvable import POM: Failure to transfer org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE from https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced. Original error: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:1.5.8.RELEASE from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/1.5.8.RELEASE/spring-boot-starter-parent-1.5.8.RELEASE.pom @ org.eclipse.sw360:resource-server:${revision}, /sw360/rest/resource-server/pom.xml, line 239, column 25
   [ERROR] 'dependencies.dependency.version' for org.springframework.security.oauth:spring-security-oauth2:jar is missing. @ org.eclipse.sw360:resource-server:${revision}, /sw360/rest/resource-server/pom.xml, line 105, column 21
   [ERROR] 'dependencies.dependency.version' for org.springframework.security:spring-security-jwt:jar is missing. @ org.eclipse.sw360:resource-server:${revision}, /sw360/rest/resource-server/pom.xml, line 116, column 21
   [ERROR] 'dependencies.dependency.version' for org.springframework.data:spring-data-rest-hal-browser:jar is missing. @ org.eclipse.sw360:resource-server:${revision}, /sw360/rest/resource-server/pom.xml, line 139, column 21
   [ERROR] 'dependencies.dependency.version' for org.springframework.security:spring-security-test:jar is missing. @ org.eclipse.sw360:resource-server:${revision}, /sw360/rest/resource-server/pom.xml, line 186, column 21
   @ 
   [ERROR] The build could not read 3 projects -> [Help 1]

-> this happens two times on resetted docker and with docker 1.9.X and 2.5.0 for mac

I have no clue why this happens, as I have built master on Monday. It really mysterious, as the error:

[ERROR]   The project org.eclipse.sw360:log4j-osgi-support:11.1.0-SNAPSHOT (/sw360/libraries/log4j-osgi-support/pom.xml) has 1 error
[ERROR]     Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:4.2.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:4.2.1: Could not transfer artifact org.apache.felix:maven-bundle-plugin:pom:4.2.1 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/4.2.1/maven-bundle-plugin-4.2.1.pom: Unknown host repo.maven.apache.org: Temporary failure in name resolution -> [Help 2]

should not happen because earlier dependencies were loaded, obviously.

[mcjaeger]

adding to this, executing the URL in the browser works https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/1.5.8.RELEASE/spring-boot-starter-parent-1.5.8.RELEASE.pom as well as the two previous maven artifacts downloaded before ...

[oheger-bosch]

This is indeed strange. The mvn commands should use the same approach regarding proxy handling and repository access in all containers. (At least, I can spot no difference between the backend and rest images.)

Are you behind a proxy? The README describes how to set a --build-arg that defines the proxy.

[mcjaeger]

Are you behind a proxy? The README describes how to set a --build-arg that defines the proxy.

no, not all. tested on direct DSL at home...

[oheger-bosch]

Does it make a difference if you build the image manually using docker? Just cd in the rest folder and execute something like docker build -t sw360/rest .

[mcjaeger]

sorry, I cannot test it.

On two other machines I got the same error. However, I found out that just simply restarting it a couple of times help. Then I have the same (restarting it several times helps) at:

Step 2/22 : RUN apt-get update && apt-get install -y curl unzip
 ---> Using cache
 ---> d6f01811c39a
Step 3/22 : COPY portal-bundle.properties portal-bundle.properties
 ---> Using cache
 ---> c5beba008726
Step 4/22 : COPY setenv.sh setenv.sh
 ---> Using cache
 ---> 209550b310d1
Step 5/22 : COPY prepare-liferay.sh prepare-liferay.sh
 ---> Using cache
 ---> f5140f12213c
Step 6/22 : RUN bash prepare-liferay.sh
 ---> Running in 6adb13695479
... download of MD5 checksum
... expecting checksum 34904d3aae3b60a658189b5b513e738c
... start downloading liferay-ce-portal-tomcat-7.3.3-ga4-20200701015330959.tar.gz (this can take some time)
... got checksum 34904d3aae3b60a658189b5b513e738c
...start downloading 3rd party dependencies
    commons-codec-1.12.jar...ERROR: Service 'sw360base' failed to build : The command '/bin/sh -c bash prepare-liferay.sh' returned a non-zero code: 6

Now the docker build ran through, but docker compose up fails with several error in the log:

sw360liferay_1         | 2020-12-07 12:45:49.756 INFO  [main][DialectDetector:159] Using dialect org.hibernate.dialect.PostgreSQLDialect for PostgreSQL 11.10
sw360postgres_1        | 2020-12-07 12:45:49.782 UTC [73] ERROR:  relation "release_" does not exist at character 61
sw360postgres_1        | 2020-12-07 12:45:49.782 UTC [73] STATEMENT:  select mvccVersion, schemaVersion, buildNumber, state_ from Release_ where releaseId = 1
sw360postgres_1        | 2020-12-07 12:45:49.817 UTC [73] ERROR:  relation "release_" does not exist
sw360postgres_1        | 2020-12-07 12:45:49.817 UTC [73] STATEMENT:  alter table Release_ add mvccVersion bigint default 0 not null
sw360postgres_1        | 2020-12-07 12:45:49.820 UTC [73] ERROR:  relation "release_" does not exist
sw360postgres_1        | 2020-12-07 12:45:49.820 UTC [73] STATEMENT:  alter table Release_ add schemaVersion varchar(75) null
sw360postgres_1        | 2020-12-07 12:45:49.821 UTC [73] ERROR:  relation "release_" does not exist at character 61
sw360postgres_1        | 2020-12-07 12:45:49.821 UTC [73] STATEMENT:  select mvccVersion, schemaVersion, buildNumber, state_ from Release_ where releaseId = 1
sw360liferay_1         | 2020-12-07 12:45:49.822 INFO  [main][DBInitUtil:173] Create tables and populate with default data

or

sw360couchdb_1         | [error] 2020-12-07T12:46:13.465149Z nonode@nohost emulator -------- Error in process <0.8285.0> with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk... 
sw360couchdb_1         | [notice] 2020-12-07T12:46:18.480104Z nonode@nohost <0.327.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:403) <= mem3_shards:load_shards_from_disk/1(line:378) <= mem3_shards:load_shards_from_disk/2(line:407) <= mem3_shards:for_docid/3(line:91) <= fabric_doc_open:go/3(line:38) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:187) <= chttpd_auth_cache:listen_for_changes/1(line:134)

I guess my lack of docker experience is not good for testing this. maybe someone else can pick it up ...

[mcjaeger]

Even after some attempts on three computers I cannot get it running. Attached the log from my latest trials. docker-test-error.log

Maybe some one else can test this successfully on a non-mac environment.

After the entire startup executing ./startUp.sh, nginx return 502 when calling https://localhost:8443/

[oheger-bosch]

@mcjaeger, thank you for testing and providing the log file. I went through the logs, but cannot find some obvious reason for what is going wrong. The containers seem to come up and are able to communicate. For instance, there is a lot of interaction between the thrift backend and couch db. Also, couch db has obviously correctly been setup: there are some messages about missing system databases, but those appear no longer after the thrift container has been initialized.

Around line 1100 in the log, couch db seems to have a problem (CRASH REPORT). I haven't seen this message before and have no idea what could be the reason for it. This is probably the cause for the exception in the thrift container starting in line 1103: here a service gets a timeout when accessing the database, causing the initialization of a Spring bean to fail.

Later on, the liferay container is starting in the usual way and initializing its OSGi bundles. This is independent on the thrift backend, as at this point of time there is no interaction with the backend yet. I have the impression that you tried to access the UI too early; the liferay container has not yet been fully started - this explains the 502 response from nginx. The log ends before the final startup message from liferay is printed. (However, as there seems to be an issue with couch db, it is unlikely that the system is working.)

[mcjaeger]

I think the setup on macos with docker is fragile, for example, starting it one time yields:

mch00830@MACC02Z14EYLVDL v2 % sudo ./startUp.sh 
Password:
Creating v2_sw360couchdb_1 ... error

ERROR: for v2_sw360couchdb_1  Cannot create container for service sw360couchdb: mkdir /var/lib/docker/image/overlay2/layerdb/mounts/8e7fe4ec5f02eb2ad888db78520f19d5a68406a77431464473355605e0b46725: bad message

ERROR: for sw360couchdb  Cannot create container for service sw360couchdb: mkdir /var/lib/docker/image/overlay2/layerdb/mounts/8e7fe4ec5f02eb2ad888db78520f19d5a68406a77431464473355605e0b46725: bad message
ERROR: Encountered errors while bringing up the project.

then starting again brings:

mch00830@MACC02Z14EYLVDL v2 % sudo ./startUp.sh
Password:
Creating v2_sw360couchdb_1 ... done
Creating v2_start_dependencies_run ... done
Waiting for sw360couchdb to listen on 5984...
sleeping
sleeping
sleeping
sleeping
sleeping
nc: bad address 'sw360couchdb'
sleeping
nc: bad address 'sw360couchdb'
sleeping
nc: bad address 'sw360couchdb'
sleeping
nc: bad address 'sw360couchdb'
sleeping
nc: bad address 'sw360couchdb'
sleeping
``

[mcjaeger]

I think it will be better to test it on a non-mac but linux machine ... if I restart the build of the containers is brings now:

Step 7/7 : CMD ["couchdb"]
 ---> Using cache
 ---> a9d7b82aa564

[Warning] One or more build-args [sw360_tag] were not consumed
Successfully built a9d7b82aa564
ERROR: Service 'sw360couchdb' failed to build : sync /var/lib/docker/image/overlay2/imagedb/metadata/sha256/a9d7b82aa5648dd06fa155d0d202a2ff245c23598779c03b6186e35b7292f800/.tmp-lastUpdated705204760: structure needs cleaning
mch00830@MACC02Z14EYLVDL v2 % 

[mcjaeger]

ugh, maybe I got it now. On macos, the docker has 2GB of RAM by default settings, I suspect that this contributes to the problem I have, the docker engine runs with 2GB of RAM only which sounds like it is not enough RAM. Will do a last test.

[mcjaeger]

Hm, my computer gets significantly slower now, but the breakups still exists, after passing some maven download errors by retrying, I am not retrying at :

 ---> Running in c543c4802066
... download of MD5 checksum
... expecting checksum 34904d3aae3b60a658189b5b513e738c
... start downloading liferay-ce-portal-tomcat-7.3.3-ga4-20200701015330959.tar.gz (this can take some time)
... got checksum 34904d3aae3b60a658189b5b513e738c
...start downloading 3rd party dependencies
    commons-codec-1.12.jar...ERROR: Service 'sw360base' failed to build : The command '/bin/sh -c bash prepare-liferay.sh' returned a non-zero code: 6

[mcjaeger]

something runs. I am not sure if the RAM setting for Docker Desktop made the trick. I think what made the trick is to use a specific tag:

sudo docker-compose build --build-arg sw360_tag=sw360-12.0.0-M1

-> it is only successful with this tag, not with current master.

[mcjaeger]

Important notice to all who look here: I only got it running by retrying building the containers several times. (like 5 -10 times or so), because of strange download issues during building the containers. Once the containers are built, the .7startUp.sh script seems to work fine.