[Enhancement]: Are you open to a PR for MongoDB?

[RXWatcher]

Describe the feature/enhancement

We love ABS but it's having issues with performance on over 100K audio books. Our goal is 300K and there is no way ABS will handle that quantity in its current form. We feel that the DB used is likely a large bottleneck. If we were to supply code to move everything to mongodb, would you be open to accepting the PR?

We can fork and go out on our own but that 1) Doesnt help the community 2) We'd have to continue to merge your chances into our repo. I dont think either is ideal so I'd rather the mongodb code go into the public repos.

I feel this is a win-win for both parties.

Please let me know your thoughts

[advplyr]

The DB is something we have been talking about upgrading for a while. The hesitation with MongoDB would be its ability to be embedded/portable. Did you have a method in mind for implementing MongoDB?

[RXWatcher]

I'll have to think about that. Mongodb came to mind because of it's no-sql structure and being fairly straight forward when moving from json files. Let me give it some thought but I appreciate you not slamming the door on the idea.

[advplyr]

I like MongoDB and it would be great if this was an embeddable option because it would save a lot of time migrating.

The bottleneck of the current db is likely because we are loading most of the data into memory on server startup. If we switch to a standard db solution then we would fetch that data when necessary instead of filling up memory. For small/medium sized libraries the current db will be faster then other standard dbs because it is just reading/writing to a file, no indexing. There is a bigger problem with the current db in that if a file write gets interrupted then it can result in data loss.

I tested another no-sql db that was built to be embedded (LiteDB) and the performance was not good. My last search for embedded no-sql dbs was about a year ago so maybe something has come along.

I realized we will probably need to setup a relational model and use sqlite like everyone else. With that we could eventually offer users the ability to use their own external db instead of the embedded one. I'm still open to anything here though.

[rasmuslos]

I would suggest using knex to handle data. This way the end user could decide for themselves what database they want to use. ABS could come preconfigured with SQLite but with an easy way to setup Postgres for big libraries like this one. Knex supports PostgreSQL, MariaDB, SQLite, and some more.

[RXWatcher]

Yeah, ideally Id want postgres but thought the leap from the current database to a sql database was too much of an ask.

[advplyr]

I'm very familiar with sequelize. I just looked through knex and it looks nice. If you have used both and can point out any pros/cons that could be helpful.

[PassMeTheMustard]

I'll admit to being unfamiliar with nosql databases but I would have thought an actual SQL database like SQLLite would be well suited to ABS instead of all the text files everywhere. Even better if there is an option for a more powerful external SQL database. What advantages would Mongodb add over that?

[RXWatcher]

The reason I suggested mongodb was that its fairly straightforward to move from the current architecture to it and I was looking for something that wouldn't throw people off. It sounds like people might be ok moving to a sql based database so mongo is a moot point. I like the option to default to SQLite but then also support postgres for larger implementations.

On Wed, Mar 8, 2023, 02:27 PassMeTheMustard @.***> wrote:

I'll admit to being unfamiliar with nosql databases but I would have thought an actual SQL database like SQLLite would be well suited to ABS instead of all the text files everywhere. Even better if there is an option for a more powerful external SQL database. What advantages would Mongodb add over that?

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/1584#issuecomment-1459125888, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLOXCPGAL7UK4OBJNX3XOTW27OBHANCNFSM6AAAAAAVQM3NJU . You are receiving this because you authored the thread.Message ID: @.***>

[advplyr]

In hindsight I would have started with SQLite and an ORM.

Abs started as a single web page serving a single folder of audiobooks where the only data needing to persist was the scanned in audiobooks. It made more sense at the time to just dump these JSON objects into a file.

I would have rolled over to a more powerful NoSQL database a while ago if I found one that was performant and embeddable. That would have been quick and resolve issues with the current db while we plan a much bigger migration to an ORM down the road. Since that doesn't seem like an option now I think the way forward is integrating an ORM, designing the data model, making sure we can get SQLite binaries working for all installs, then writing a migration file.

[advplyr]

I pushed a branch that uses Sequelize ORM and sqlite.

https://github.com/advplyr/audiobookshelf/tree/sqlite

I tested building for docker, debian and the windows exe (not in use yet). The database gets created in /config/database.sqlite and I setup one User model to test. Running the server will just connect to the db and create a User if it doesn't exist.

The builds will need to be tested on different systems still. Next up would be building the data model.

[rasmuslos]

I would suggest using knex to handle data. This way the end user could decide for themselves what database they want to use. ABS could come preconfigured with SQLite but with an easy way to setup Postgres for big libraries like this one. Knex supports PostgreSQL, MariaDB, SQLite, and some more.

Sorry for the late reply... I think they are more or less the same, so i dosen't really matter. A major reason for me to use knee were the automatic migrations, but sequelize has them too. As far as I know the only difference is the way the query builder works, I prefer the approach knee takes, but the idea is fundamentally the same, so again, it doesn't really matter