resource watcher, log watcher, libcloud, object_storage

[vlerkin]

As a separate thread, resource watcher was added. When a new pod with specific labels (job labels) are aded to the cluster and are in a running state, another watcher starts to follow it's logs and write it to a file stored on a managing pod with scrapyd. Then all files are uploaded to an object storage of a preferred provider, which is implemented via Apache Libcloud library. Persistent Volume and PVC are commented out in the Kubernetes yaml file and were not tested in the cloud, but it can be used to add them to the managing pod to store log files in a persistent volume.

[wvengen]

And regarding naming: I would propose to call this feature joblogs (or maybe job logging).

I think that ideally we'd move all files related to this feature to a subdirectory joblogs, which would include object storage (if we would start using object storage in another component as well, then we would move it to its own subdirector). In that way, all things related to this feature can be found there, and as a developer that is working on the code, you can easily ignore it if you don't want to dive into this.

[vlerkin]

Thank you for your feedback!

So if you manage, would you be able to use a class for the Kubernetes joblogs implementation?

I do agree with your reasoning and I will work on that, also a great moment for me to learn a bit more about design patterns that may be beneficial for this case.