search

MarkGaox / Dune

All-in-one data management platform that allows you to govern your data infrastructures more easily.
0 stars 1 forks source link

Log + Monitor + Other Fault-tolerant Features #2

Closed MarkGaox closed 2 years ago

MarkGaox commented 2 years ago

For the logging part, Heroku will capture any standard output and aggregate them as app log. So no much work to do. Please just make sure all important information are generated in the log, for example, task failure, task success, permission deny, and so on. If you found any existing logs output is unnecessary, feel free to delete it.

reference: https://devcenter.heroku.com/articles/logging

Please also focus on the monitoring and other fault-tolerant features, for example, if we encountered 400 error or 500 error during tasks execution, then what should do? Should we re-execute the whole list of tasks or adaptively re-execute some tasks. It's all up to you to decide what to do. If you have any idea, feel free to discuss with me.

MarkGaox commented 2 years ago

Optional: Caching

MarkGaox commented 2 years ago

According to Junyu, monitor works correctly: image

MarkGaox commented 2 years ago

We decided to not move forward with load balancing part since Heroku doesn't reveal its public IP address.

Junyu Update: Bought and configured a load balancing middleware with public IP from Tecent Cloud

MarkGaox commented 2 years ago

Caching is also discarded.