[Score + Minio] - Storage client error

MitchellShiell commented 4 months ago

Summary

Score is failing on startup with Error creating bean with name 'activeStorageProfile' defined in class path resource [bio/overture/score/server/config/ProfileConfig.class]

Description

I created an issue10 branch for inspection and reproduction (steps below).
The docker compose referenced in this issue is linked here.

The current Object Storage setup uses Minio as follows

  minio:
    image: quay.io/minio/minio
    container_name: minio
    ports:
      # For S3 requests
      - "9000:9000"
      # For Minio Console
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: admin123
    command: server /data --console-address ":9001"
    volumes:
      - ./persistentStorage/data-minio:/data
    networks:
      - backend-network

The Minio console can be access from localhost:9001/login

[!IMPORTANT]
Based on a comment within an older Score docker-compose there may be compatibility issues. The image used in the referenced compose is no longer available.

A minio Client image creates an object and state bucket

  minio-client:
    image: minio/mc
    container_name: minio-client
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://host.docker.internal:9000 admin admin123 &&
      /usr/bin/mc mb myminio/state &&
      /usr/bin/mc mb myminio/object;
      exit 0;
      "

No issues here

Then Score fails to spin up with the following configuration, Full error log is in the comment below

  score:
    image: ghcr.io/overture-stack/score-server:47f006ce
    container_name: score
    platform: linux/amd64
    depends_on:
      minio-client:
        condition: service_completed_successfully
      keycloak:
        condition: service_started
      song:
        condition: service_started
    ports:
      - "8087:8087"
    environment:
      # Spring Variables
      SPRING_PROFILES_ACTIVE: amazon,aws,prod,secure,jwt
      SERVER_PORT: 8087
      # Song Variable
      METADATA_URL: song:8080
      # Score Variables
      UPLOAD_PARTSIZE: 1073741824
      UPLOAD_CONNECTION_TIMEOUT: 1200000
      SERVER_SSL_ENABLED: "false"
      # Minio Variables
      S3_ENDPOINT: http://localhost:9000
      S3_ACCESSKEY: admin
      S3_SECRETKEY: admin123
      S3_SIGV4ENABLED: "true"
      OBJECT_SENTINEL: heliograph
      BUCKET_NAME_OBJECT: object
      BUCKET_NAME_STATE: state
      BUCKET_SIZE_POOL: 0
      BUCKET_SIZE_KEY: 2
      # Keycloak Variables
      AUTH_SERVER_PROVIDER: keycloak
      AUTH_SERVER_CLIENTID: score
      AUTH_SERVER_CLIENTSECRET: scoresecret
      AUTH_SERVER_TOKENNAME: apiKey
      AUTH_SERVER_KEYCLOAK_HOST: keycloak:8080
      AUTH_SERVER_KEYCLOAK_REALM: myrealm
      AUTH_SERVER_SCOPE_STUDY_PREFIX: STUDY.
      AUTH_SERVER_SCOPE_DOWNLOAD_SUFFIX: .READ
      AUTH_SERVER_SCOPE_DOWNLOAD_SYSTEM: score.WRITE
      AUTH_SERVER_SCOPE_UPLOAD_SYSTEM: score.READ
      AUTH_SERVER_SCOPE_UPLOAD_SUFFIX: .WRITE
      AUTH_JWT_PUBLICKEYURL: http://keycloak:8080/oauth/token/public_key
      SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: http://keycloak:8080/realms/myrealm/protocol/openid-connect/certs
      # Logging Variables
      LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB: INFO
      LOGGING_LEVEL_BIO_OVERTURE_SCORE_SERVER: INFO
      LOGGING_LEVEL_ROOT: INFO
    networks:
      - backend-network

[!NOTE] From Leo: "Score doesn’t need to set AUTH_JWT_PUBLICKEYURL instead it uses SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI set to http://keycloak:8080/realms/myrealm/protocol/openid-connect/certs

Open question: what the heck is OBJECT_SENTINEL: heliograph

Steps to Reproduce

Clone the following Repo

git clone -b issue10 https://github.com/overture-stack/composer.git

Run the Container

docker compose up -d

Score will fail in this configuration however for testing I will leave the Song and Score Client setup commands for this configuration below. Persistent storage here should allow you to use the same API key, if not you will need to generate one from the stage UI and update it here (user admin password admin123).

Song Client

docker run -d -it --name song-client \
-e CLIENT_ACCESS_TOKEN=8a5591ec-92e9-4bd4-bbc0-4c2da6652e53 \
-e CLIENT_STUDY_ID=SECA-CA \
-e CLIENT_SERVER_URL=http://localhost:8080 \
--network="host" \
--mount type=bind,source=./demoData/data,target=/output \
[ghcr.io/overture-stack/song-client:5.1.1](http://ghcr.io/overture-stack/song-client:5.1.1)

Score Client

docker run -d -it --name score-client \
    -e CLIENT_ACCESS_TOKEN=82e57f53-c23b-4392-9b77-4a75ce9a0730 \
    -e STORAGE_URL=http://localhost:8087 \
    -e METADATA_URL=http://localhost:8080 \
    --network="host" \
    --mount type=bind,source="$(pwd)",target=/output \
    ghcr.io/overture-stack/score:5.10.0

[!TIP] A previously generated manifest.txt ready for upload can be found at ./demoData/data/manifest.txt if the storage volumes are working correctly you should see Song is already storing a few unpublished analyses.

MitchellShiell commented 4 months ago

Score container logs

Output from Scores docker container

2024-06-12 13:33:44.980  INFO 7 --- [           main] b.overture.score.server.config.S3Config  : master key id : 'af628f04-ac12-4b11-bf83-6545fd44ad18'
2024-06-12 13:33:44.982  INFO 7 --- [           main] b.overture.score.server.config.S3Config  : Using AWSS3V4SignerType
2024-06-12 13:33:47.307  INFO 7 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-06-12 13:33:47.723  WARN 7 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'activeStorageProfile' defined in class path resource [bio/overture/score/server/config/ProfileConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.lang.String]: Factory method 'activeStorageProfile' threw exception; nested exception is java.util.NoSuchElementException: No value present
2024-06-12 13:33:47.738  INFO 7 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2024-06-12 13:33:47.775  INFO 7 --- [           main] ConditionEvaluationReportLoggingListener : 
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2024-06-12 13:33:47.837 ERROR 7 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'activeStorageProfile' defined in class path resource [bio/overture/score/server/config/ProfileConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.lang.String]: Factory method 'activeStorageProfile' threw exception; nested exception is java.util.NoSuchElementException: No value present
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.18.jar!/:5.3.18]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.18.jar!/:5.3.18]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) ~[spring-boot-2.6.6.jar!/:2.6.6]
        at bio.overture.score.server.ServerMain.main(ServerMain.java:30) ~[classes!/:5.10.1-SNAPSHOT]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[score-server.jar:5.10.1-SNAPSHOT]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[score-server.jar:5.10.1-SNAPSHOT]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[score-server.jar:5.10.1-SNAPSHOT]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[score-server.jar:5.10.1-SNAPSHOT]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [java.lang.String]: Factory method 'activeStorageProfile' threw exception; nested exception is java.util.NoSuchElementException: No value present
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.3.18.jar!/:5.3.18]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.18.jar!/:5.3.18]
        ... 27 common frames omitted
Caused by: java.util.NoSuchElementException: No value present
        at java.base/java.util.Optional.get(Unknown Source) ~[na:na]
        at bio.overture.score.server.config.ProfileConfig.activeStorageProfile(ProfileConfig.java:21) ~[classes!/:5.10.1-SNAPSHOT]
        at bio.overture.score.server.config.ProfileConfig$$EnhancerBySpringCGLIB$$855f4b77.CGLIB$activeStorageProfile$0(<generated>) ~[classes!/:5.10.1-SNAPSHOT]
        at bio.overture.score.server.config.ProfileConfig$$EnhancerBySpringCGLIB$$855f4b77$$FastClassBySpringCGLIB$$bd911727.invoke(<generated>) ~[classes!/:5.10.1-SNAPSHOT]
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.3.18.jar!/:5.3.18]
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.3.18.jar!/:5.3.18]
        at bio.overture.score.server.config.ProfileConfig$$EnhancerBySpringCGLIB$$855f4b77.activeStorageProfile(<generated>) ~[classes!/:5.10.1-SNAPSHOT]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.18.jar!/:5.3.18]
        ... 28 common frames omitted

MitchellShiell commented 4 months ago

Solution

Updated MINIO container as follows

  # ==========================
  # Minio
  # ==========================
  minio:
    image: minio/minio:RELEASE.2018-05-11T00-29-24Z
    container_name: minio
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY: admin
      MINIO_SECRET_KEY: admin123
    command: server /data
    volumes:
      - ./persistentStorage/data-minio:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  # ==========================
  # Minio Client
  # ==========================
  minio-client:
    image: minio/mc
    container_name: minio-client
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://host.docker.internal:9000 admin admin123 &&
      /usr/bin/mc mb myminio/state &&
      /usr/bin/mc mb myminio/object &&
      /usr/bin/mc mb myminio/object/data;
      exit 0;
      "

[!NOTE]

The minio-client is not necessary but it does mean no manual folder set up if they are not already in the repo for you, this will be removed in the future.

minio/minio:RELEASE.2018-05-11T00-29-24Z uses ACCESS KEY and SECRET KEY which may be more compatible with score

There was also a note in scores docker compose which went as follows #NOTE: must use 2018-05-11 version, since the ETag rules are different. 2018 will return the part md5, anything newer will not, and score-server expects etag==md5

I also updated the score container with the collaboratory profile, and updated the S3_ENDPOINT with the localhost url (as opposed to the docker url minio:9000 which won't work with the swagger).

  score:
    image: ghcr.io/overture-stack/score-server:47f006ce
    container_name: score
    platform: linux/amd64
    depends_on:
      keycloak:
        condition: service_started
      song:
        condition: service_started
    ports:
      - "8087:8087"
    environment:
      # Spring Variables
      SPRING_PROFILES_ACTIVE: collaboratory,prod,secure,jwt
      SERVER_PORT: 8087
      # Song Variable
      METADATA_URL: http://song:8080
      # Score Variables
      SERVER_SSL_ENABLED: "false"
      # Minio Variables
      S3_ENDPOINT: http://localhost:9000
      S3_ACCESSKEY: admin
      S3_SECRETKEY: admin123
      S3_SIGV4ENABLED: true
      # Test object for pinging
      OBJECT_SENTINEL: heliograph
      BUCKET_NAME_OBJECT: object
      BUCKET_NAME_STATE: state
      UPLOAD_PARTSIZE: 1073741824
      UPLOAD_CONNECTION_TIMEOUT: 1200000

Why this works I do not know, but would like to, need to spend more time looking at these profiles, however they are currently being removed/updated from legacy project namings.
By creating a dummy file named heliograph the directory object/data/heliograph I am able to successfully ping the score server using the score swagger endpoint:

The highlighted url takes me, as expected to the heliograph file with my customized success message

MitchellShiell commented 4 months ago

There is a maybe related score-client issue now, I will open a new ticket and link it here

overture-stack / composer