search

dbpedia / databus

A digital factory platform for managing files online with stable IDs, high-quality metadata, powerful API and tools for building on data: find, access, make interoperable, re-use
Apache License 2.0
41 stars 17 forks source link

User data somehow vanished - now deploy is not working anymore #26

Closed yum-yab closed 1 year ago

yum-yab commented 1 year ago

Problem

Currently it is not possible to submit any data to the energy databus for me and the other users, not by webinterface or the API. It seems to work for new users though (see here). So it seems to happen if you have an older account (exact date of not working anymore is not known, but probably 2-4 weeks before).

The error is:

curl -H "X-API-Key: REDACTED" -H "Content-Type: application/json" -X PUT https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27 -d '{"@context": "https://downloads.dbpedia.org/databus/context.jsonld", "@graph": [{"@id": "https://energy.databus.dbpedia.org/denis/general", "@type": "Group", "title": "Group Title", "abstract": "Some Abstract text", "description": "Some description text"}, {"@type": "Dataset", "@id": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27#Dataset", "hasVersion": "2022-09-27", "title": "Test Title", "abstract": "This a short abstract for the dataset. Since this is only a test it is quite insignificant.", "description": "A bit longer description of the dataset.", "license": "http://this.is.a.license.uri.com/test", "distribution": [{"@id": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27#testartifact_type=ontology_topic=pizza.owl", "@type": "Part", "file": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27/testartifact_type=ontology_topic=pizza.owl", "formatExtension": "owl", "compression": "none", "downloadURL": "https://yum-yab.github.io/data/databus-api-test/first/pizza-ont.owl", "byteSize": 163302, "sha256sum": "b71592685053db4171d2eaacb1eb1084d927fd6831f7ee8c9329903f90c72763", "dcv:type": "ontology", "dcv:topic": "pizza"}, {"@id": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27#testartifact_type=randomData_topic=persons.csv", "@type": "Part", "file": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27/testartifact_type=randomData_topic=persons.csv", "formatExtension": "csv", "compression": "none", "downloadURL": "https://yum-yab.github.io/data/databus-api-test/first/Sample500.csv", "byteSize": 34243, "sha256sum": "1c69ff99c105ab0f3459a4cd928f14284c996702148a2f62637df69f3e1a01ab", "dcv:type": "randomData", "dcv:topic": "persons"}, {"@id": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27#testartifact_type=turbineData_extra=external.json", "@type": "Part", "file": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27/testartifact_type=turbineData_extra=external.json", "formatExtension": "json", "compression": "none", "downloadURL": "https://openenergy-platform.org/api/v0/schema/supply/tables/wind_turbine_library/rows/", "byteSize": 173469, "sha256sum": "56476b9c4130f931e164fdee2fe11ba275f30eff9046317013cbcffebd1c6922", "dcv:type": "turbineData", "dcv:extra": "external"}, {"@id": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27#testartifact_type=randomData_topic=persons_extra=different.csv.bz2", "@type": "Part", "file": "https://energy.databus.dbpedia.org/denis/general/testartifact/2022-09-27/testartifact_type=randomData_topic=persons_extra=different.csv.bz2", "formatExtension": "csv", "compression": "bz2", "downloadURL": "https://yum-yab.github.io/data/databus-api-test/first/Sample500.csv.bz2", "byteSize": 11059, "sha256sum": "247f4458ea39bc4d689226980749a766132688f88b241912144a8ff225623b48", "dcv:type": "randomData", "dcv:topic": "persons", "dcv:extra": "different"}]}]}'
Publishing DataId.
> 47 triples selected via construct query.
> Auto-completed the input.
> SHACL validation successful.
> Publishing as "https://energy.databus.dbpedia.org/denis#this".
> No signature found in the input.
> Generating signature.
> Failed to generate signature. Please contact an administrator.
Publishing DataId finished with code 500.

Expected behavior

It should be possible to deploy data and the data should not get lost.

Further documentation

On the energy databus somehow data in the gstore got lost. This can be related to the fancy docker container mashup @JJ-Author did in the past, but im not sure.

On a new request (just as above) gstore logs give the following error:

13:12:57.055 WARN  ROOT - org.dbpedia.databus.swagger.api.DefaultApi: Error processing request
java.nio.file.NoSuchFileException: /gstore/git/denis/webid.jsonld
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.Files.newByteChannel(Files.java:361)
    at java.nio.file.Files.newByteChannel(Files.java:407)
    at java.nio.file.Files.readAllBytes(Files.java:3152)
    at org.dbpedia.databus.LocalGitClient.$anonfun$readFile$2(GitClient.scala:59)
    at scala.util.Try$.apply(Try.scala:209)
    at org.dbpedia.databus.LocalGitClient.$anonfun$readFile$1(GitClient.scala:59)
    at org.dbpedia.databus.LocalGitClient.wrapWithSync(GitClient.scala:112)
    at org.dbpedia.databus.LocalGitClient.readFile(GitClient.scala:59)
    at org.dbpedia.databus.ApiImpl.readFile(ApiImpl.scala:118)
    at org.dbpedia.databus.ApiImpl.getFile(ApiImpl.scala:53)
    at org.dbpedia.databus.swagger.api.DefaultApi.$anonfun$new$6(DefaultApi.scala:121)
    at org.scalatra.ScalatraBase.liftAction(ScalatraBase.scala:280)
    at org.scalatra.ScalatraBase.$anonfun$invoke$1(ScalatraBase.scala:274)
    at org.scalatra.ApiFormats.withRouteMultiParams(ApiFormats.scala:183)
    at org.scalatra.ApiFormats.withRouteMultiParams$(ApiFormats.scala:169)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRouteMultiParams(DefaultApi.scala:32)
    at org.scalatra.ScalatraBase.invoke(ScalatraBase.scala:273)
    at org.scalatra.ScalatraBase.invoke$(ScalatraBase.scala:272)
    at org.dbpedia.databus.swagger.api.DefaultApi.org$scalatra$json$JsonSupport$$super$invoke(DefaultApi.scala:32)
    at org.scalatra.json.JsonSupport.$anonfun$invoke$1(JsonSupport.scala:90)
    at org.scalatra.ApiFormats.withRouteMultiParams(ApiFormats.scala:183)
    at org.scalatra.ApiFormats.withRouteMultiParams$(ApiFormats.scala:169)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRouteMultiParams(DefaultApi.scala:32)
    at org.scalatra.json.JsonSupport.invoke(JsonSupport.scala:84)
    at org.scalatra.json.JsonSupport.invoke$(JsonSupport.scala:83)
    at org.dbpedia.databus.swagger.api.DefaultApi.invoke(DefaultApi.scala:32)
    at org.scalatra.ScalatraBase.$anonfun$runRoutes$3(ScalatraBase.scala:248)
    at scala.Option.flatMap(Option.scala:171)
    at org.scalatra.ScalatraBase.$anonfun$runRoutes$1(ScalatraBase.scala:246)
    at scala.collection.immutable.Stream.flatMap(Stream.scala:486)
    at org.scalatra.ScalatraBase.runRoutes(ScalatraBase.scala:245)
    at org.scalatra.ScalatraBase.runRoutes$(ScalatraBase.scala:243)
    at org.dbpedia.databus.swagger.api.DefaultApi.runRoutes(DefaultApi.scala:32)
    at org.scalatra.ScalatraBase.runActions$1(ScalatraBase.scala:169)
    at org.scalatra.ScalatraBase.$anonfun$executeRoutes$6(ScalatraBase.scala:181)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at org.scalatra.ScalatraBase.cradleHalt(ScalatraBase.scala:199)
    at org.scalatra.ScalatraBase.executeRoutes(ScalatraBase.scala:181)
    at org.scalatra.ScalatraBase.executeRoutes$(ScalatraBase.scala:150)
    at org.dbpedia.databus.swagger.api.DefaultApi.executeRoutes(DefaultApi.scala:32)
    at org.scalatra.ScalatraBase.$anonfun$handle$1(ScalatraBase.scala:123)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at org.scalatra.DynamicScope.withResponse(DynamicScope.scala:79)
    at org.scalatra.DynamicScope.withResponse$(DynamicScope.scala:77)
    at org.dbpedia.databus.swagger.api.DefaultApi.withResponse(DefaultApi.scala:32)
    at org.scalatra.DynamicScope.$anonfun$withRequestResponse$1(DynamicScope.scala:59)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at org.scalatra.DynamicScope.withRequest(DynamicScope.scala:70)
    at org.scalatra.DynamicScope.withRequest$(DynamicScope.scala:68)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRequest(DefaultApi.scala:32)
    at org.scalatra.DynamicScope.withRequestResponse(DynamicScope.scala:58)
    at org.scalatra.DynamicScope.withRequestResponse$(DynamicScope.scala:56)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRequestResponse(DefaultApi.scala:32)
    at org.scalatra.ScalatraBase.handle(ScalatraBase.scala:123)
    at org.scalatra.ScalatraBase.handle$(ScalatraBase.scala:119)
    at org.dbpedia.databus.swagger.api.DefaultApi.org$scalatra$servlet$ServletBase$$super$handle(DefaultApi.scala:32)
    at org.scalatra.servlet.ServletBase.handle(ServletBase.scala:53)
    at org.scalatra.servlet.ServletBase.handle$(ServletBase.scala:46)
    at org.dbpedia.databus.swagger.api.DefaultApi.org$scalatra$servlet$FileUploadSupport$$super$handle(DefaultApi.scala:32)
    at org.scalatra.servlet.FileUploadSupport.handle(FileUploadSupport.scala:95)
    at org.scalatra.servlet.FileUploadSupport.handle$(FileUploadSupport.scala:80)
    at org.dbpedia.databus.swagger.api.DefaultApi.org$scalatra$CorsSupport$$super$handle(DefaultApi.scala:32)
    at org.scalatra.CorsSupport.$anonfun$handle$1(CORSSupport.scala:221)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at org.scalatra.DynamicScope.withResponse(DynamicScope.scala:79)
    at org.scalatra.DynamicScope.withResponse$(DynamicScope.scala:77)
    at org.dbpedia.databus.swagger.api.DefaultApi.withResponse(DefaultApi.scala:32)
    at org.scalatra.DynamicScope.$anonfun$withRequestResponse$1(DynamicScope.scala:59)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
    at org.scalatra.DynamicScope.withRequest(DynamicScope.scala:70)
    at org.scalatra.DynamicScope.withRequest$(DynamicScope.scala:68)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRequest(DefaultApi.scala:32)
    at org.scalatra.DynamicScope.withRequestResponse(DynamicScope.scala:58)
    at org.scalatra.DynamicScope.withRequestResponse$(DynamicScope.scala:56)
    at org.dbpedia.databus.swagger.api.DefaultApi.withRequestResponse(DefaultApi.scala:32)
    at org.scalatra.CorsSupport.handle(CORSSupport.scala:209)
    at org.scalatra.CorsSupport.handle$(CORSSupport.scala:206)
    at org.dbpedia.databus.swagger.api.DefaultApi.handle(DefaultApi.scala:32)
    at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:65)
    at org.scalatra.ScalatraServlet.service$(ScalatraServlet.scala:64)
    at org.dbpedia.databus.swagger.api.DefaultApi.service(DefaultApi.scala:32)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:856)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:531)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:319)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:175)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:133)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:754)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:672)
    at java.lang.Thread.run(Thread.java:748)
holycrab13 commented 1 year ago

This might have been caused by some changes to the database. However we should explore how we can make the environment more robust

Options:

kurzum commented 1 year ago

Closing this, quite old...