search

meituan / ptubes

Ptubes is a database disaster recovery product based on the PITR (Point In Time Recovery) method, which can be used to restore the entire database to a specific point in time to help users improve the reliability and security of the database
Other
54 stars 11 forks source link

[Feature]It is recommended that ptubes support the separation of storage and computation #6

Open myksl opened 2 years ago

myksl commented 2 years ago

Describe the feature

I recently encountered a problem. Our business has some tasks with relatively large traffic, which will write a large amount of data. At the same time, because of business characteristics, we expect the data to be stored for a long time, which will cause our disk usage to be relatively high. In extreme cases will cause disk exhaustion So, I hope to have a new ability

  1. Added a new StorageMode inside Reader, which supports S3 remote storage + Memory mode, and supports tiered storage
  2. Support Binlog persistent storage
  3. The real-time consumption stream maintains the same speed as before
  4. The local disk will be used as the real-time data layer, S3 will be used as the persistent storage layer, and the non-latest data of the local disk will be asynchronously written to S3 through certain rules