search

ubiquity-os-marketplace / generate-vector-embeddings

0 stars 6 forks source link

Enable RLS #2

Open gentlementlegen opened 3 weeks ago

gentlementlegen commented 3 weeks ago

The database can be accessed by anyone public with a full read / write access, which raises security concerns. RLS should be enabled on all the tables and a token required to modify data.

0x4007 commented 3 weeks ago

It would be nice to do git based storage but not sure if it is feasible for vectors given the amount of data that needs to be saved.

sshivaditya2019 commented 3 weeks ago

The database can be accessed by anyone public with a full read / write access, which raises security concerns. RLS should be enabled on all the tables and a token required to modify data.

I can work on this. Are you looking to allow write access with user_role ? or more like authenticated user with supabase ?

sshivaditya2019 commented 3 weeks ago

It would be nice to do git based storage but not sure if it is feasible for vectors given the amount of data that needs to be saved.

I just made some rough calculations, we could create a repository with sqlite and store close to 150,000 entries with the same schema as the supabase one. This is assuming Git-LFS has a free plan limit of 2GB.

0x4007 commented 3 weeks ago

I can work on this. Are you looking to allow write access with user_role ? or more like authenticated user with supabase ?

Does anybody have a rough time estimate for this project?

I just made some rough calculations, we could create a repository with sqlite and store close to 150,000 entries with the same schema as the supabase one. This is assuming Git-LFS has a free plan limit of 2GB.

Assuming on average 50 comments daily, we get a bit over 8 years of storage. Perhaps git based storage is a good idea!

sshivaditya2019 commented 3 weeks ago

I can work on this. Are you looking to allow write access with user_role ? or more like authenticated user with supabase ?

Does anybody have a rough time estimate for this project?

That should depend on how fine grained you want the control on write-access.

I just made some rough calculations, we could create a repository with sqlite and store close to 150,000 entries with the same schema as the supabase one. This is assuming Git-LFS has a free plan limit of 2GB.

Assuming on average 50 comments daily, we get a bit over 8 years of storage. Perhaps git based storage is a good idea!

The Data Update and Data Creation would be hassle. So, application would require an in-memory staging database, after certain number of commits, they would forward the changes to the db.

0x4007 commented 3 weeks ago

@gentlementlegen or @rndquu perhaps you can add a time estimate, and can elaborate on

That should depend on how fine grained you want the control on write-access.