Does Depot allow for storing relational data in Redis?

[rlam3]

Does Depot allow for storing data in Redis instead but still be able to use SQLAlchemy models to retrieve the data?

Meaning... How do I use SQLAlchemy Model to retrieve the data of the image uploaded to S3. But have the data on Redis?

I'm asking because I'm under the impression that it is costly in the long run to store a read-only data structure in a postgres environment. It can be done, but is it necessary?

Thanks!

[vlcinsky]

@rlam3 in short, I think it is not possible.

In detail:

There different kinds of storages:

• database (accessible via Sqlalchemy): great for relational data, some fields may refer to named data blobs stored in depot (regardless of how the blob is stored)
• redis is great for storing simple data structures (hashes, keys, sets, lists etc), but in general does not allow relational queries and as it stores data in memory, there is practical limit in amount of data which can be stored there.
• s3, local file system etc. are just means to store named data blobs.

So to your questions:

• does depot allow for storing data in Redis?: Currently there is no such middleware. One very similar one is existing in-memory middleware.
• ... but still be able to use SQLAlchemy models to retrieve data? Yes, it would allow it. But be aware, that the query would not have any means to decide on attributes hidden in stored data blobs.
• Meaning...: The second paragraph sounds like you expect relational operations on data in Redis. Not really possible. Definitely not via SQLAlchemy. This is against nature of Redis storage.

You are right, that storing (larger amounts) of read-only data blobs in relational database is often not very efficient. This is exactly where depot shines. And if you really need to do searches based on blob content, you wold have to deploy some sort of indexing (it would be very close to full text search solution).

My conclusions: Do not expect simple queries based on attributes hidden inside depot managed blobs.

Redis as middleware would be very similar to existing in-memory model with advantages such as persistence and scaling.

[rlam3]

@vlcinsky Thank you for your input. I believe there was some misunderstanding in my comment above. What I would like to re-emphasize is the use of storing read-only data in Postgres doesn't make sense. Efficiency wise...

I'd like to know if there was a better way to store the .file/ metadata for the path of the url/sizes/etc... instead through a large blob inside postgres table/cell. This is where I was thinking of using Redis or some NOSQL to store this data and have a reference back to SQLALCHEMY model.

The file currently goes straight to S3. This is not what I'm worried about.

Thanks.

[amol-]

I'm not sure I got the benefit that you hope to achieve by storing value of the file metadata if you had to load the model pointing to those files anyway {as you are talking about accessing it through sqlalchemy it means the model was queried).

Also I'm not sure I got the reference to read only data, file metadata is not read-only, it changes whenever you change the file.