Ingestion service that fetches data from endpoint at regular intervals

[alex855k]

📋 User story

As a system I want a service that can ingest data from an endpoint at regular intervals from the latest cached data entry point, so I can update the data in a cache database with the latest logs entries.

Description

To divide some responsibilities onto smaller service, it's necessary to have a small scale-able service, that is able to request data on a daily interval(3AM CET +0100) from Mono Solutions endpoint, ingest the data. And send it to the correct queue in at RabbitMQ. This will be done using a docker image of .net core.

Prerequisites

• [x] Find an appropiate .net core docker image and configure it to run with our source code data.

Acceptance Criteria

• [x] Create LogSource class with logic for which endpoints to connect to, the payload; credentials and format of the data. Additionally it needs to have a list of queue names where the payload will be send.
• [x] Create a crontab that will use LogSource to run the job, every 24th hour starting at 3AM.
• [x] Create the class will send the envelope object to the appropriate queues.
• [x] Log errors and actions
• [x] When the user story is done host it in a docker hub repository and put the repository data in /CSProjectEAL/GraBID main github repository in a seperate folder.
• [x] Unit tests.

Additional Comments

In it's current state you cant run solution before refiner service has been fixed. So hold off on the review.

child #6

[IulianBogdan]

Testing

[alex855k]

Reopened