search

distributed-system-analysis / pbench

A benchmarking and performance analysis framework
http://distributed-system-analysis.github.io/pbench/
GNU General Public License v3.0
188 stars 108 forks source link

Convert archive from file-system based architecture to an object-store based architecture #1789

Open portante opened 4 years ago

portante commented 4 years ago

To allow us to scale up various operational aspects of the Pbench Server, we need to replace the file-system based architecture with one based on an object-store.

All the "ARCHIVE" tree should be moved to S3, with the notion of tracking operations to be performed via symlinks replaced by records in some sort of database (sql or no-sql stores).

npalaska commented 4 years ago

I wonder if we can also move the results directory contents to S3 since they are also static in nature. however, that would mean dashboard has to make an api call to s3 everytime it wants to list the contents on dashboard (not sure if it is even possible to do so with s3), but anyway its just a thought came to my mind while thinking if we should move away from storing any user sensitive data on the server.

dbutenhof commented 4 years ago

The idea is that the tarballs will be in S3. If we need persistent access to unpacked files, we'll need to figure out a unique name mapping scheme (e.g., user.controller..file, maybe) to store them in S3 as well. We'll probably want the ToC of the tarball in a DB rather than reading from S3.

And we definitely want all access encapsulated through pbench server APIs; the front end shouldn't touch (or know about) S3 any more than it should be directly accessing our Elasticsearch server.