Integration server - work status

[Baeist]

Overview

New features	Description	Status	Comments
Clean up Ndrive	We want to eventually delete the imported files after they have been persisted in ERDA	-	-
Clean up local files	Files are copied into a local directory from the ndrive. We want to delete those after they have been persisted with ERDA	-	-
Close ars open share	Closing an open file share once the files have been downloaded to hpc.	-

Robustness	Description	Status	Comments
Connection with slurm	Currently each microservice that communicates with slurm just connects once when initated. This could cause issues since its assumed that this connection is continually live. At the same time im not convinced we want to start up a new connection for each command we want to send.	-	-
Better checks for datafields	This relates to the fields that are put into databases.	-	Ensuring that each field contains only what it is allowed to and in the correct format would be good. Would help prevent issues of wrong inputs before it reaches the integration server.
Flexibility for datafields	This relates to the fields that are put into databases.	-	Being able to accept a changing set of data and converting it into the correct format for storage.

Error handling	Description	Status	Comments
Keycloak auth failure	How do we handle if keycloak does not authenticate or is down.	-	There are two main problems here: first one is what we do with the whole pipeline, should we shut everything down while we wait for keycloak. Secondly what do we do with assets that are directly affected by this. This issue is generic for all services that make use of ARS.
ARS create asset fails	When attempting to create an asset through the ars endpoint something goes wrong	-	It could fail for multiple reasons. Wrong datatypes, formatting, server fail, graphdb fail - we only get the message that it failed, so we need to handle this generically. I think putting in an error status in the track database for is in ars would work. Then we can set up logging and a message system around that.
ARS file uploader no metadata	After finding an asset with files that should be uploaded to ars, we need to get some information from the corresponding metadata- if this for some reason is not obtained we should handle that	-	Possible this is too much error handling, since it should be impossible for there not to be corresponding metadata. We could handle this by setting the has_new_files and files_status to error and then set up logging for this and a message system.
ARS file uploader file path not found	When checking the file path for the file we are looking to upload the file is not there.	-	We could handle this by setting the has_new_files and files_status to error and then set up logging for this and a message system.
ARS file uploader upload fails	When attempting to upload the file through the ars api we dont get a status 200 back.	-	If we get a 507 back that would tell us that the crc doesnt match and we can try again, then if that fails again with a 507 we could set has_new_files and files_status to error and then set up logging for this and a message system. If we get any other type of status error we can handle that in the same way
ARS sync erda fail	When requesting the sync something unfortunate happens.	-	It is purely ARS that fails here. We could handle this by setting the sync_erda and files_status to error and then set up logging for this and a message system
ARS erda sync validation ERDA_ERROR	This error appears when ars is not able to sync the asset with erda.	-	When this happens then issue lies with erda. We could handle this by setting a timer and then retrying. Changing the status of erda_sync to retry or similar and then set up logging for this and a message system. We could also consider setting a general delay on validation process since its very likely that whatever caused this issue will be ongoing for some time. Maybe set up our own erda status listener that could then tell when to restart again.
ARS erda sync validation api fail	Our call to the ars api failed	-	This is the worst scenario. We have an asset that has been successfully requested to sync and now we cant get any status updates for it. We just dont know what happened after the sync request. If this happens i suggest we stop most/all processes until it resolves and we get confirmation on what is happening. Maybe implement a dry run that can check that everything is up and running again before continuing. We can set erda_sync to error and log this and send the information somewhere.
ARS update metadata not in ars	If for some reason the asset has not yet been created in ars its metadata cant be updated.	-	I dont remember why this could happen (maybe it cant). Maybe something with timing or derivatives. We can solve this by setting update_metadata to error and then log it and send the information somewhere.
-issue	-description	-status	-comment
-issue	-description	-status	-comment
-issue	-description	-status	-comment

Scalability	Description	Status	Comments
Docker containers	Running multiple of each microservice would alleviate any concerns of causing bottlenecks from the integration server. Would also be easier to make changes and deploy each service.	-	-

Optimization	Description	Status	Comments
None	We dont need anything	Perfection	Great job

Configuration and deployment	Description	Status	Comments
Databse setup	Some standardized way of setting up the mongodb.	-	-
API deployment	Description of the needed server configuration for running the rest endpoint.	-	-

Documentation	Description	Status	Comments
Deploymnet guide	A guide on how to setup and run each microservice, the mongodb and the api endpoints.	-	-
-README for documentation	The documentation part of the integration servers repository is growing and a readme would help make it more manageable	-	-
Overall README	Should contain updated information on the project.	-	More explanation is required about each module and their function.

| -issue | -description | -status | -comment |