Broker does not start when database is empty

[sebplorenz]

After cleaning all databases and doing a fresh start of the Broker version registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/core:5.0.2 and registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/fuseki:5.0.2 I'm getting the following broker log:

23-11-2022 08:28:41.576 [main] INFO  de.fraunhofer.iais.eis.ids.index.common.persistence.RepositoryFacade.getNewReadOnlyConnectionToFuseki - retry to establish connection to Fuseki server in 5 seconds
 23-11-2022 08:28:46.586 [main] INFO  de.fraunhofer.iais.eis.ids.index.common.persistence.RepositoryFacade.getNewReadOnlyConnectionToFuseki - unable to establish a connection to Fuseki server with url http://mobids-fuseki:3030/connectorData/sparql
 23-11-2022 08:28:46.587 [main] INFO  de.fraunhofer.iais.eis.ids.index.common.persistence.RepositoryFacade.getNewReadOnlyConnectionToFuseki - stop trying to establish connection and throw exception ... 
Nov 23, 2022 8:28:46 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Tomcat]
 08:28:46.623 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'main': Invocation of init method failed; nested exception is HttpException: 404 Not Found

Obviously the Broker cannot connect to the fuseki with resource or schema connectorData

[timwirtz86]

@sebplorenz can you please provide us with a reproducible setup? In the best-case use this Postman collection

[sebplorenz]

I cannot use a Postman Collection since the Broker, that this collection is talking to, is not starting up!

This is my setup. It was working without problems as long as there was content in the database. Now, after cleaning the database, it is not starting up anymore:

version: '2'
services:

  broker-core:
    image: registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/core:5.0.1-MDS
    restart: always
    volumes:
      - /app/broker/cert:/etc/cert
    environment:
      - SPARQL_ENDPOINT=http://mobids-fuseki:3030/connectorData
      - SPARQL_URL=http://mobids-fuseki:3030/connectorData
      - ELASTICSEARCH_HOSTNAME=mobids-elasticsearch
      - SHACL_VALIDATION=true
      - DAPS_VALIDATE_INCOMING=true
      - COMPONENT_URI=https://broker.mobilitydataspace.io
      - COMPONENT_CATALOGURI=https://broker.mobilitydataspace.io/connectors/
      - SSL_JAVAKEYSTORE=/etc/cert/ids.broker.mobilitydataspace.io.p12
    expose:
      - "8080"
    ports:
      - "8080:8080"
    links:
      - back-end:backend

  mobids-fuseki:
    image: registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/fuseki:5.0.2
    restart: always
    volumes:
      - broker_mobids-fuseki:/fuseki
    expose:
      - "3030"
    ports:
      - "3030:3030"

  mobids-elasticsearch:
    image: elasticsearch:7.16.2
    restart: always
    environment:
      - http.port=9200
      - http.cors.enabled=true
      - http.cors.allow-origin=https://catalog.mobilitydataspace.io #Change the Value
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - http.cors.allow-credentials=true
      - discovery.type=single-node
    volumes:
      - broker_mobids-esdata:/usr/share/elasticsearch/data
    expose:
      - "9200"
    ports:
      - "9200:9200"

  mobids-searchfrontend:
    image: registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/frontend:5.0.1
    restart: always
    expose:
      - "80"
    ports:
      - "80:80"
    environment:
      - "REACT_APP_BROKER_URL=https://catalog.mobilitydataspace.io"

  mongodb:
    image: mongo:latest
    restart: always
    volumes:
      - broker_db:/data/db

  back-end:
    image: registry.gitlab.cc-asp.fraunhofer.de/eis-ids/broker/mongodb-handler:4.2.7
    restart: always
    expose:
      - "4000"
    ports:
      - "4000:4000"
    environment:
      - MONGODB_ENDPOINT=mongodb://mongodb:27017/users
      - JWT_SECRET=anewstring
      - ADMIN_PASSWORD=xyz
      - BROKER_URL=http://broker-core:8080
      - LOG4J_FORMAT_MSG_NO_LOOKUPS=true
    links:
      - mongodb:mongo

volumes:
  broker_mobids-fuseki:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /app/broker/db/fuseki
  broker_mobids-esdata:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /app/broker/db/esdata
  broker_db:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /app/broker/db/broker

[sebplorenz]

Hab Dir geantwortet.

Von: Tim Wirtz @.> Gesendet: Mittwoch, 23. November 2022 09:35 An: International-Data-Spaces-Association/metadata-broker-open-core @.> Cc: Lorenz, Sebastian @.>; Mention @.> Betreff: Re: [International-Data-Spaces-Association/metadata-broker-open-core] Broker does not start when database is empty (Issue #126)

@sebplorenzhttps://github.com/sebplorenz can you please provide us with a reproducible setup? In the best-case use this Postman collectionhttps://github.com/Mobility-Data-Space/MDS-Broker-Core/blob/main/doc/IDS%20Broker%20ES%20Tests%20with%20MDS.postman_collection.json

— Reply to this email directly, view it on GitHubhttps://github.com/International-Data-Spaces-Association/metadata-broker-open-core/issues/126#issuecomment-1324706149, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA7YBCJO2W4MYD3O4YT6Q73WJXJLLANCNFSM6AAAAAASIV57UA. You are receiving this because you were mentioned.Message ID: @.***>

[timwirtz86]

In summary:

1. you had a running instance of the Broker
2. Connectors and resources have been successfully added and/or removed
3. then you cleaned the Fuseki
4. Next time the Broker was connecting to the Fuseki it didn't finds it

How did you exactly clean the Fuseki?

[sebplorenz]

I shut down the setup, removed everything from the docker volume directories and started it all up again. As if it would be started on a new server.

[timwirtz86]

to be sure you mean that you (manuelly) remove the content of the folder

• /app/broker/db/fuseki
• /app/broker/db/esdata
• /app/broker/db/broker

Why not using: https://docs.docker.com/engine/reference/commandline/compose_down/ is says that Stops containers and removes containers, networks, volumes, and images created by up. would that be what you want you want to achieve?

[sebplorenz]

Yes, I removed these folders. That is not the problem. The problem is, that the Broker does not startup now since the databases are empty. What can be the reason that a request to http://mobids-fuseki:3030/connectorData/sparql is responded with a 404?

[timwirtz86]

@sebplorenz, no idea 🙈. I never had that. Í cleaned my setup either with Docker-compose down or using the fuseki UI by removing and insert again the data set. The elastic was cleaned by curl -X DELETE 'http://localhost:9200/_all'. We discussed it internally and believe that "simply" removing the volumnes might cause problems due acitons docker is supposed to take when it initializes the containers.

Could you please try with a fresh setup: either Docker-compose down or my manual approach?

[timwirtz86]

@sebplorenz does the proposed solution work for you ?