Introduced Express to run as server so that this can live in Kubernetes as a long running server, express providing a /health endpoint for the liveness probe.
Since we now have express, I added two more routers:
/jobs path for checking the cron job status, and activating/deactivating the cron job
/etl path for initiating the ETL immediately. Use /etl/all to run all stages or /etl/extract, /etl/transform , /etl/load for just a single stage.
The application still runs as a command line app now accepting two new flags:
--cron will start it as an application that will trigger the ETL (all stages) on a schedule defined in the env.
--server will start the express server, which allows the ETL to be started via the above web api.
The Dockerfile has been updated to run the application with both cron and server flags.
Introduced Express to run as server so that this can live in Kubernetes as a long running server, express providing a /health endpoint for the liveness probe.
Since we now have express, I added two more routers:
/jobspath for checking the cron job status, and activating/deactivating the cron job/etlpath for initiating the ETL immediately. Use/etl/allto run all stages or/etl/extract,/etl/transform,/etl/loadfor just a single stage.The application still runs as a command line app now accepting two new flags:
--cronwill start it as an application that will trigger the ETL (all stages) on a schedule defined in the env.--serverwill start the express server, which allows the ETL to be started via the above web api.The Dockerfile has been updated to run the application with both cron and server flags.