search

FAIRDataTeam / FAIRDataPoint

https://www.fairdatapoint.org
MIT License
59 stars 37 forks source link

FAIRDataPoint unstable: breaking at the first restart #170

Open vemonet opened 3 years ago

vemonet commented 3 years ago

Describe the bugs

A few months ago we started a FAIRDataPoint in "production", we logged in, created users, changed the weird default users/passwords manually, and we even added datasets metadata. It was "working" (the form to add metadata was really simple and did not help the user in adding dataset metadata, and the whole thing was not user-friendly to use, but it was at least showing some text fields for the datasets we entered!)

This was a few months ago, now when we go to the FAIRDataPoint we get greeted with 404, see by yourself at https://fairdatapoint.semanticscience.org/

The only error showing in the docker logs is:

fdp_1         | 2021-07-23 10:44:25,145 56916 [http-nio-80-exec-4] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requesed url https://fairdatapoint.semanticscience.org
fdp_1         | 2021-07-23 10:44:25,187 56958 [http-nio-80-exec-4] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - No metadata found for the uri 'https://fairdatapoint.semanticscience.org'

I am trying to connect with the login/password I gave at the time, but I am getting bad login, and I can't reset the password

To Reproduce Steps to reproduce the behavior:

  1. Start a FAIRDataPoint server with docker-compose
  2. Change the login/password
  3. It should stop working for no reason at some point and show a 404 on the main page

Expected behavior

Multiple things should be improved to make the FAIRDataPoint more stable and production-ready:

  1. Remove the default albert.einstein user/password, this is really bad practice for service that are expected to go to productions at multiple places. This should be defined as environment variables at the start of the docker-compose (like for most services with login). So that the admin can at least reconnect
  2. Implement OAuth login instead of requiring user to create a new account with a new password. It would make complete sense to use ORCID OAuth here or SOLID authentication, worst case you can also use GitHub, GitLab, Google or even Facebook if you want, but if you want to create an API based on Linked Data principle, it does not make sense to create another silo of user data, and ask your users to create another account (note that it is easy to implement OAuth with Spring, easier than creating and managing a whole user base)

Context Please fill the following and eventually add additional information (e.g. about used storage in case that issue is storage-related):

Here is the docker-compose.yml we use:

version: '3'
services:
    fdp:
        image: fairdata/fairdatapoint:1.6.0
        volumes:
            - ./application.yml:/fdp/application.yml:ro
    fdp-client:
        image: fairdata/fairdatapoint-client:1.6.0
        environment:
            FDP_HOST: fdp
            VIRTUAL_HOST: fairdatapoint.semanticscience.org
            LETSENCRYPT_HOST: fairdatapoint.semanticscience.org
        ports:
            - 8081:80
    mongo:
        image: mongo:4.0.12
        volumes:
            - /data/fairdatapoint/mongo/data:/data/db
    blazegraph:
        image: metaphacts/blazegraph-basic:2.2.0-20160908.003514-6
        volumes:
            - /data/fairdatapoint/blazegraph:/data

I already reported this issue when I started with FDP: https://github.com/FAIRDataTeam/FAIRDataPoint/issues/94

At the time I somehow managed to get it running by tweaking permissions of the Blazegraph volumes, but this wasn't a stable fix

Note that FDP can be deployed with multiple triplestores as backend, I used Blazegraph because this was the one pushed in the "production deployment" documentation: https://fairdatapoint.readthedocs.io/en/latest/deployment/production-deployment.html

So I was expecting blazegraph to be the triplestore working the best with FDP, but it does not seems to be the case

Maybe we should use another triplestore? But which one? It would be nice to have a clear idea of the exact stack that needs to be setup for production (with working persistent volumes)

vemonet commented 2 years ago

Hi @kburger ,

It seems like more people like @xiaofengleo are having this problem: https://github.com/FAIRDataTeam/FAIRDataPoint/issues/94#issuecomment-894743763

Is there any plan on fixing this major issue that prevents to reuse your service seriously in production?

Not sure what's the goal of this "FAIRDataPoint" service and the "FAIRDataTeam", but seems like this is a service that should be easy to deploy at various places, to make the world more FAIR probably.

Unfortunately the current stack relies on a database that can break and lose all data at any moment (which is the last thing you want when you are trying to build a trustful catalog of datasets.)

kburger commented 2 years ago

Hi @vemonet, I'm sorry to hear about these issues. Unfortunately, I don't see the same behaviour for the demo instances we are running. The underlying systems are a bit different (running on Ubuntu with GraphDB), but that should not make that big of a difference. Can you provide me with some system specs? CPU and RAM for example?

AndreaFurlani commented 5 months ago

Hello, actually me and my team are having the same issue. After the first restart we get this exact same error and even by reinstalling the FDP we get the exact same error. We are working on a remote server VM running linux ubuntu 5.4.0 and last version of FDP with GraphDB.

Docker-compose file was not changed, we changed only variables.scss to change icon and colours and application.yml to use our VM IP address (clientUrl: http://VM_IP:7070).

dennisvang commented 1 month ago

Seeing similar status 404 issues for path / after restart, running locally on Windows 10 Home (Docker on WSL2) and also on Raspbian.

Using a basic docker compose file configured for port 8080, as described in the docs, with FDP version 1.17:

application.yml

instance:
  clientUrl: http://localhost:8080

compose.yml

version: '3'
services:
  fdp:
    image: fairdata/fairdatapoint:1.17
    volumes:
      - ./application.yml:/fdp/application.yml:ro
  fdp-client:
    image: fairdata/fairdatapoint-client:1.17
    ports:
      - 8080:80
    environment:
      - FDP_HOST=fdp
  mongo:
    image: mongo:4.0.12

Steps to reproduce:

  1. docker compose up (optional -d)
  2. browse to http://localhost:8080, and so on, to verify everything works
  3. docker compose stop (wait for it to finish)
  4. docker compose start (again wait)
  5. browse to http://localhost:8080 or http://localhost:8080/my-metadata to see a status 404 not found
  6. other paths, like /users or /swagger-ui/index.html, do work

Here's some logging from fdp-1:

...
2024-07-22 21:43:53 2024-07-22 19:43:53,281 933041 [http-nio-80-exec-8] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/actuator/info
2024-07-22 21:43:57 2024-07-22 19:43:57,038 936798 [http-nio-80-exec-3] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/resource-definitions
2024-07-22 21:44:24 2024-07-22 19:44:24,456 964216 [http-nio-80-exec-10] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/
2024-07-22 21:44:24 2024-07-22 19:44:24,459 964219 [http-nio-80-exec-6] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/spec
2024-07-22 21:44:24 2024-07-22 19:44:24,469 964229 [http-nio-80-exec-2] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/meta
2024-07-22 21:44:24 2024-07-22 19:44:24,470 964230 [http-nio-80-exec-10] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - No metadata found for the uri 'http://localhost:8080'
2024-07-22 21:44:24 2024-07-22 19:44:24,486 964246 [http-nio-80-exec-2] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - No metadata found for the uri 'http://localhost:8080'
2024-07-22 21:44:38 2024-07-22 19:44:38,433 978193 [http-nio-80-exec-7] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/users
2024-07-22 21:44:52 2024-07-22 19:44:52,759 992519 [http-nio-80-exec-4] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/metadata-schemas
2024-07-22 21:44:58 2024-07-22 19:44:58,149 997909 [http-nio-80-exec-1] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/settings
2024-07-22 21:45:02 2024-07-22 19:45:02,961 1002721 [http-nio-80-exec-9] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://localhost/dashboard
2024-07-22 21:45:02 2024-07-22 19:45:02,966 1002726 [http-nio-80-exec-9] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Original requested url http://localhost/dashboard
2024-07-22 21:45:02 2024-07-22 19:45:02,967 1002727 [http-nio-80-exec-9] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requested url http://localhost:8080/dashboard
2024-07-22 21:45:02 2024-07-22 19:45:02,983 1002743 [http-nio-80-exec-9] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - No metadata found for the uri 'http://localhost:8080'

There's another issue that also looks very similar:

However, that issue was (presumably) caused by Blazegraph, whereas we use the in-memory store (no persistence).