search

SciCatProject / scicat-filewriter-ingest

Python client that connects to a kafka queue and creates new datasets when receiving notification that a file has been written
0 stars 0 forks source link

Ingestor testing environment #17

Open nitrosx opened 2 months ago

nitrosx commented 2 months ago

Currently the ingestor is tested on the production environment. As it is a critical component of the ESS FAIR data system, we need to set up a CI/CD pipeline. Automated testing is the first step towards it. The next image should be a good representation at high level of what is needed for testing. SciCat_ingestor_testing_environment

YooSunYoung commented 2 months ago

@nitrosx I'm not sure if you want a monitor or a test from this.

YooSunYoung commented 2 months ago

We can probably do sth similar to ECDC integration tests

https://gitlab.esss.lu.se/ecdc/ess-dmsc/kafka-to-nexus/-/blob/main/.gitlab-ci.yml?ref_type=heads#L151-179

YooSunYoung commented 2 months ago

We can probably do sth similar to ECDC integration tests

https://gitlab.esss.lu.se/ecdc/ess-dmsc/kafka-to-nexus/-/blob/main/.gitlab-ci.yml?ref_type=heads#L151-179

I tried to download the same docker-compose file but I had this error in the CI so I'll just copy+paste the relevant parts...

Run docker-compose up -d

yaml.scanner.ScannerError: mapping values are not allowed here
  in "./docker-compose.yml", line 10, column 17
Error: Process completed with exit code 1.