Logging with Elastic stack

[aninda052]

This pr will close #86

Crete a docker-compose file for running Elastic Stack. The container will run 4 services -

• Elasticsearch - For storing data
• APM - For tracking application performence
• Filebeat - For Shipping log to elasticsearch
• Kibana - For visualization

To write application log on log file, we use @elastic/ecs-winston-format package for formatting log into ECS format, this format help filebeat to parse log properly and also this package add a extra trace id along with other data with each log

[foyzulkarim]

Amazing work @aninda052 I am really feeling proud 🥳

• Please add a documentation for the future users to help them how to run the ElasticSearch (eg. Username, password, where to see the logs etc.). You can add a separate readme file inside of the EKA folder and reference it in the main readme.
• Please add a condition so that we can spin up the elasticsearch according to our preference (I assume the ES will be running everytime)

[aninda052]

• Please add a condition so that we can spin up the elasticsearch according to our preference (I assume the ES will be running everytime)

@foyzulkarim vai, can you tell me more details about the condition? i already add a env variable which will be responsible for activating APM Node Agent.

[foyzulkarim]

For example, in our code we have integrated Loggly, but what if people don't want to use Loggly? As a condition, I am checking like this if (process.env.ENABLE_WINSTON_LOGGLY === 'true') {

So, if anyone doesn't enable loggly, we will not run loggly specific configuration. Thus system will not beak.

I suggest you to make the ElasticSearch configurable in this way so that if we don't enable the ES specific thing, the code will not even run the ES. Also, may be we can create another npm command to run with ES like set ES=true && npm run es && npm start

Above suggestions are just for an idea, feel free to tweak them as you see fit. 🥳

[aninda052]

I suggest you to make the ElasticSearch configurable in this way so that if we don't enable the ES specific thing, the code will not even run the ES.

@foyzulkarim vai, our system doesn't depend on elastics stack. i made all the elastic stack related docker configuration in a different directory with separate readme so that we can run elastics stack on a different container. Apart from APM Node Agent there is no other ES specific config in our system codebase. APM Node Agent is used for sending http related log/performance to elasticsearch. This agent can be active by a environment variable called IS_APM_ACTIVE. By any chance the value of IS_APM_ACTIVE is true and the elasticsearch is not running, then our system will not break.

Also, may be we can create another npm command to run with ES like set ES=true && npm run es && npm start

i like the idea of running ES Stack by a npm command.