search

EricTheMagician / DriveFS

A google drive fuse filesystem implemented in C++
Mozilla Public License 2.0
63 stars 7 forks source link

PSA: Migrating to Postgresql #9

Open EricTheMagician opened 5 years ago

EricTheMagician commented 5 years ago

For anyone actively using DriveFS, it will be migrated to postgres from mongodb in the coming weeks.

BerriJ commented 5 years ago

Interesting. Why so?

Is it for performance?

Thanks a lot for all the effort you are putting into this Project.

EricTheMagician commented 5 years ago

It was originally motivated by performance and memory issues. MongoDB on my system was a huge memory hog, and that was something I wanted to tackle by switching databases.

The other was to reduce overall memory usage, especially for very large amounts files/folders by not having all the required information in memory, and falling back to the database, when needed.

As for performance, I did expect a small loss compared to having the data stored in memory, but it's quite significant. I primarily use it through rclone and I have noticed it hanging from time to time with ls. By hanging, it would last a few seconds, sometimes alot more, on the order of a minute / did I code this incorrectly moments. fixed, performs reasonably well.

mcadam commented 5 years ago

Would this work with stateless container if everything is in database now and not in memory anymore? As in if I run multiple times the program connecting to the same database it would work? I am running it in a cluster to mount the drive in multiple places and currently have multiple databases so would be nice to cut on that :)

EricTheMagician commented 5 years ago

It's still not stateless as some temporary states are still being stored. I still have some occasional crashes, so I want to fix those before pushing it to the master branch.

I'm currently traveling for work, and so I won't have too much time to work on this until august. So stay tuned.

mcadam commented 5 years ago

Hi, any updates on that one? :)