search

medizininformatik-initiative / GeMTeX

7 stars 0 forks source link

Access of the annotated data at inception-gemtex-deid-base_project-grascco_raw #17

Closed AbanoubAbdelmalak closed 6 months ago

AbanoubAbdelmalak commented 6 months ago

Hello,

I would like to ask how to read or visualize the annotation files ending with .ser .

Thank you.

reckart commented 6 months ago

I assume the question is in relation to INCEpTION. The .ser files used by INCEpTION are not meant for external consumption.

If you want to access the annotated data, choose UIMA CAS JSON or UIMA CAS XMI (XML 1.0) as export format on the annotation page or secondary export format in the project export panel. Both of these formats are can be read and written by the DKPro Cassis Python library as well as the Apache UIMA Java SDK using either the org.apache.uima.util.CasIOUtils class or UIMA JSON CAS IO library. The JSON and XML formats are also human-readable. But they are standoff annotation formats, not inline formats, so the annotations are encoded separately from the text inside the files.

That said, you can read the .ser files e.g. using methods from the org.apache.uima.util.CasIOUtils class or using the DKPro Core BinaryCasReader component.

AbanoubAbdelmalak commented 6 months ago

I have tried to import the project into INCEpTION but it did not respond and no error message was generated. I am using INCEPTION version 31.4. I have tried to import some of the test projects on INCEPTION example page and it worked perfectly.

reckart commented 6 months ago

Do you have access to the logs?

AbanoubAbdelmalak commented 6 months ago

I am running it on a container on Kubernetes cluster. When looking at the logs of this container, there were no trace of me trying to import this project. I could find logs of me importing other projects successfully but no trace of this one (not even error messages). 

Part of the logs: 
2024-04-23 16:41:23 INFO [SYSTEM] ProjectServiceImpl - Created project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] ConstraintsServiceImpl - Created constraints set [Category constraints] in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] ConstraintsServiceImpl - Saved constraints set [Category constraints] in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] ConstraintsExporter - Imported constraint [Category constraints] for project [Example: Frame annotation] with id [9]
2024-04-23 16:41:23 INFO [SYSTEM] ProjectServiceImpl - Created permission [MANAGER] for user [admin] on project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] ProjectServiceImpl - Created permission [CURATOR] for user [admin] on project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] ProjectServiceImpl - Created permission [ANNOTATOR] for user [admin] on project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] DocumentRepositoryExporter - Imported [0] document repositories for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created tagset [Argument role](27) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created 6 tags and updated 0 tags in tagset [Argument role](27) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created tagset [Frame category](28) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created 6 tags and updated 0 tags in tagset [Frame category](28) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] SourceDocumentExporter - Imported content for source document 1/1: [document.tsv](18) in [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] SourceDocumentExporter - Imported [1] source documents into aProject ([Example: Frame annotation](9))
2024-04-23 16:41:23 INFO [SYSTEM] ProjectLogExporter - Imported log for project [Example: Frame annotation] with id [9]
2024-04-23 16:41:23 INFO [SYSTEM] LoggedEventExporter - Imported [34] logged events for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] DefaultProjectPreferencesExporter - Imported [0] default preferences for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] UserProjectPreferencesExporter - Imported [0] user preferences for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] DocumentServiceImpl - Created annotation document [admin@document.tsv](5) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationDocumentExporter - Imported content for annotation document 1: user [admin] for [document.tsv](18) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationDocumentExporter - Imported content for annotation document 2: user [INITIAL_CAS] for [document.tsv](18) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationDocumentExporter - Imported [1] annotation documents for project [Example: Frame annotation] (0 seconds)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created layer [de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Lemma](36) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created feature [value](39) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created layer [de.tudarmstadt.ukp.dkpro.core.api.semantics.type.SemArg](37) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created layer [de.tudarmstadt.ukp.dkpro.core.api.semantics.type.SemPred](38) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created feature [arguments](40) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created feature [category](41) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created layer [de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token](39) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Created feature [lemma](42) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] AnnotationSchemaServiceImpl - Updated layer [de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Lemma](36) in project [Example: Frame annotation](9)
2024-04-23 16:41:23 INFO [SYSTEM] RecommenderExporter - Imported [0] recommenders for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] LearningRecordExporter - Imported [0] learning records for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] KnowledgeBaseExporter - Imported [0] knowledge bases for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] GazeteerExporter - Imported [0] gazeteeres for project [Example: Frame annotation]
2024-04-23 16:41:23 INFO [SYSTEM] ProjectExportServiceImpl - Imported project [Example: Frame annotation](9) (0 seconds)
2024-04-23 16:41:23 INFO [SYSTEM] ApplicationPageBase - admin: Project [Example: Frame annotation] successfully imported

The logs show the creation of the project Example: Frame annotation and after that nothing else although I have tried few times to import inception-gemtex-deid-base_project-grascco_raw

reckart commented 6 months ago

Ok. I have made a test locally:

Maybe try downloading the ZIP from the repo again? Make sure to not extract/repackage it.

AbanoubAbdelmalak commented 6 months ago

Thank you for testing it. I am still looking at what could be wrong with my installation. Is there any scenarios where INCEpTION would not add events to the logs?

reckart commented 6 months ago

During a project import various messages should be logged by default (see your example from above).

If none of these are logged when you try importing the template, the process fails very early.

Maybe you can contact me by mail and we can look at your instance together.

reckart commented 6 months ago

Would you like to provide a PR to the INCEpTION Kubernetes tips with a warning about not being able to import projects if the ingress configuration is not allowing a proper payload size and an example of how to increase the payload size in the configuration?

https://github.com/inception-project/inception/blob/main/inception/inception-doc/src/main/resources/META-INF/asciidoc/admin-guide/installation_kubernetes.adoc

AbanoubAbdelmalak commented 6 months ago

Would you like to provide a PR to the INCEpTION Kubernetes tips with a warning about not being able to import projects if the ingress configuration is not allowing a proper payload size and an example of how to increase the payload size in the configuration?

https://github.com/inception-project/inception/blob/main/inception/inception-doc/src/main/resources/META-INF/asciidoc/admin-guide/installation_kubernetes.adoc

Yes sure, Should I also add the setup of a MariaDB and how to connect to it by the INCEpTION service? I am a beginner at Kubernetes but I think it could be helpful to others and someone else with more experience can improve upon it.

reckart commented 6 months ago

I am not very familiar with Kubernetes either. I guess in a proper cloud environment, one would use a hosted database instead of deploying one oneself in a Kubernetes container. Describing how to set up a hosted DB for all the clouds out there doesn't seem sensible. However, tips on how to deploy a DB as a Kubernetes pod for a private deployment might help some users.