search

lsst-uk / lasair-project-management

Event handling site for LSST:UK
Apache License 2.0
9 stars 0 forks source link

Short list candidate technologies for blob store #58

Closed RoyWilliams closed 4 years ago

RoyWilliams commented 4 years ago

Based on requirements, platformed on Openstack, identify a few promising technologies. One should be standard Unix filesystem.

RoyWilliams commented 4 years ago

Here are four technologies. Maybe there are more. I wonder what our Gaia colleagues are using.

Cassandra: http://cassandra.apache.org/ Automatic replication and cluster scaling.

MinIO: https://min.io/ MinIO is High Performance Object Storage released under Apache. It is API compatible with Amazon S3.

RADOS: https://docs.ceph.com/docs/bobtail/rados/ Ceph’s RADOS Object Store is the foundation for all Ceph clusters. Being implemented by Dave Morris for ESCAPE project

Swift: https://docs.openstack.org/swift/latest/ OpenStack Swift is a highly available, distributed, eventually consistent object/blob store.

Zarquan commented 4 years ago

Being implemented by Dave Morris for ESCAPE project

Not in the plan at the moment.

RoyWilliams commented 4 years ago

Oh sorry Dave, I'm mixing up Rucio and Rados ...

RoyWilliams commented 4 years ago

Gareth Roy Terry met 5 Feb Actions:

Email on Requirements -- longevity -- data mining images requirements -- data mining light curve requirements

Management -- reliability / backups -- scalability -- cost-effective -- no hierarchy

Experment with Swift on openstack-edinburgh -- mirrormaker kafka cache -- multiple ingesters -- put images into swift -- put light curves into swift

gpfrancis commented 4 years ago

Based on Stephen's responses to the email I've updated the requirements page at https://lsst-uk.atlassian.net/wiki/spaces/LUSC/pages/865501225/Blob+Storage+Requirements and started a new page for shortlisted technologies at https://lsst-uk.atlassian.net/wiki/spaces/LUSC/pages/885227561/Blob+Storage+Technology+Shortlist