Setup sidecar container to keep log

[ssz1997]

1. Overlay is never a good place for storing logs.
2. In the case of OOM, the logs are gone. Even kubectl get logs --previous can't get the old log.

In this doc https://kubernetes.io/docs/concepts/cluster-administration/logging/, sidecar is a common way to keep the logs. We use hostPath here and the reason we add a sidecar is because not all users have the permission to log onto the host machine.

[ssz1997]

I tried two things, both of them don't have to use sidecar.

1. give enough pod memory, but not enough heap memory. After OOM, the logs inside the contains are gone, but can still be accessible by running kubectl logs <pod>. But this is not enough because this log will rotate after 10M, which is fairly small and may already lost when people find out. Unless they have another logging agent
2. not enough pod memory. The pod status will show OOM when run kubectl get po. This is different from the first case because OOM is triggered from k8s, not jvm. In this case it's easy to spot the problem and no need of logs at all.

Therefore we should use hostPath, but no need of sidecar.

[ssz1997]

@Kai-Zhang I made the hostPath configurable in case hostPath is not available cuz I don't know whether it is available. PTAL. Thanks!

[ssz1997]

The k8s pod will restart after OOM. We now have persistent log for people to do investigation.