Storing medata in SQL ?

[VannTen]

Hello o/

I came upon your library while looking for integration between S3 (well, Ceph) and sqlalchemy, and while looking through the documentation and the code I had a question:

I can see that object metadata are stored in a "sidecar" object in the object storage backend, if I read storage.py docstrings correctly. Could this metadata be stored in the SQL table instead, either as a JSON object or in separate field ? (So as to permit querying against it from the SQL side)

I'm not overly familiar with extending SQLalchemy with new fields, maybe you already considered that possibility and discarded it ?

Thanks for your work and reading me.

[jowilf]

Hello @VannTen,

The metadata sent by default to object storage backend is just the filename and content_type. You can add additional properties if you want. The important thing to know is that the File object inherit from python dict you can add whatever you want and there will be stored into your database. Just make sure to not use the default attributes used by File object internally.

You have several possibilities to add additional data:

• By using Processor or Validator
• By adding individually to each File object.

Please read the new section on the documentation for more details

Hope this help you

[VannTen]

Ok I got it (I think)

So if I resume:

The metadata key of a File object is stored in a .metadata.json suffixed document in the object storage.

The File object itself is stored as a JSON document in the database itself.

So I can already store whatever I want in the DB, and looking at the code, it should be fairly easy to sublass File and override store_content to not store the metadata.json in the object storage at all.

Many thanks for your answer :)

[jowilf]

You almost got it :)

Just a little precision The file .metadata.json is created only for local storage which did not support object metadata. You can see it clearly here

All others storage backend support metadata. Take a look at https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html

I will close this issue now. Thank you