uber-archive / cherami-server

Distributed, scalable, durable, and highly available message queue system. This project is deprecated and not maintained.

https://eng.uber.com/cherami/

MIT License

1.42k stars 102 forks source link

[ON-HOLD] Fixes around read-only storehost (on low disk space) #322

Closed kirg closed 6 years ago

kirg commented 6 years ago

notify controller when storehost is in read-only
controller would stop placing new extents on stores that are read-only
make threshold free-space based (instead of available space percentage)
improve logging on OpenAppendStream failures
(minor annoyance) rename 'storagemonitor" to "spaceMon" to reduce file completion collision on 'stor' with "storehost.go". :-)

coveralls commented 6 years ago

Coverage decreased (-0.2%) to 66.444% when pulling 91c8dea090ffc14b61d849b5400bb7239cb08b84 on storehost-readonly into 65a1bb1404b63ce07025a2a3f6d34187fddce030 on master.

thuningxu commented 6 years ago

Can we separate refactoring from the logic change? This logic is rather important to get right and reviewed throughly. Suggestion:

PR 1:

make threshold free-space based (instead of available space percentage)

PR 2:

notify controller when storehost is in read-only
controller would stop placing new extents on stores that are read-only

PR 3: rest

kirg commented 6 years ago

@thuningxu okay; will try and split up this PR.

kirg commented 6 years ago

I have created two new PRs:

325: prevents controller from placing new extents on read-only stores

326: makes storehost read-only protection based on absolute free-space, instead of percentages

I am going to leave this PR open for now. After the above two have been landed, I will remove those changes from this PR. leaving this PR to address the following, more minor issues:

improve logging on OpenAppendStream failures
(minor annoyance) rename 'storagemonitor" to "spaceMon" to reduce file completion collision on 'stor' with "storehost.go". :-)

kirg commented 6 years ago

closing out this PR, since there's too many conflicts with master, etc. will recreate PR with the rest of the changes ..