search

cloudfoundry-community / firehose-to-syslog

Send firehose events from Cloud Foundry to syslog.
MIT License
44 stars 58 forks source link

F2S Upgrade Question #199

Closed nwoike closed 6 years ago

nwoike commented 6 years ago

Not an issue, just a few questions about upgrading F2S.

We deploy Firehose to Syslog to Cloud Foundry as an application, scaling to the number of Traffic Controllers based on this guide.

We are just wondering if there are any special considerations that we should make when upgrading F2S to avoid dropping messages.

shinji62 commented 6 years ago

Hi,

First I usually do not deploy the nozzle in CF. I prefer to deploy on VM (using bosh or other related tool).

Do the Doppler buffers already protect us during the short downtime of cf push?

it will but only up to X logs... not sure the numbers this day.

Would a blue/green deployment strategy help any here? I am not sure what effect temporarily doubling the number nozzle instances here might have (assuming they would all share the same subscription id).

Will help but not resolve all your issue, the nozzle support draining on (os.Interrupt, os.Kill signal) but this do not work very well on CF.

Is there any potential for dropped logs in F2S when BOSH is upgrading the Loggregator components during a Cloud Foundry/PAS upgrade?

If VM are recreated like diego cells application will be start then stop... So yes he could happen. If only Loggregator or Doppler Vm are recreated I guess the buffer they have could be lost .. but I am not sure there, maybe they have a concept of draining too.

poy commented 6 years ago

Hey @nwoike I currently work on Loggregator!

For a CF pushed app, when you push/restart/restage it, you will lose messages. All the connections to loggregator will be cut because CF will bring down all the instances and then bring them all up.

The two ways to mitigate this are to use blue/green deployment like you suggested, or use BOSH to deploy it like @shinji62 suggested.

There is some potential for dropped logs while Loggregator rolls, but we try to prevent that with drain scripts... So normally we see minimal loss here.

nwoike commented 6 years ago

Thanks for the replies this helps clear up our questions.