new database base?

[CuddleBear92]

i see you guys use SQLLite at this point and its very limited with large number of entries and i would think that the program would work better and faster if it has another database to work from.

something like RavenDB ( http://ravendb.net/ )should work wonders as hydrus is already in a sql base.

ofc there is many others out there that could speed up the process for you guys as i find this program having issues once it has enough content to handle and it just freezes alot of the time, i want to use this program but its hard to do anything when it does freeze.

reasoning i talk about a database is well because other programs i use have seen better performance from using a full MSSQL or MySQL solution aswell as would get even better performance with something like RavenDB....... then again, i dont know anything about the code and such but i did notice it had sqllite in its folders which is a flag for me.

[hydrusnetwork]

Thank you for your input. Although I have my own complaints about SQLite, I am generally happy with its speed and capabilities working with the amounts of content hydrus typically wants it to. Most of the slowdowns in the program are due to my bad code or users' fragged/busy hard drives. SQLite is also very small and very free, which most of its competitors are not.

When does the client freeze for you?

[CuddleBear92]

in general really, when i start up it takes alot of time as its stuck Not responding in windows. (last boot used 38min before it showed more any more than a white screen) would sync folders really slow me down that much at all? the thing is that it does not seem like its using any hardware power at all when if it is slow alot of the time.

say importing a folder and downloading at the same time with the program gives me small freezes that lasts for two sec at a time which makes it hard to use. is there any limits that you have not set that you would recommend people using for a stable experience? (ofc with having a HUGE collection) the only limits i have is hardrive usage but everything else can be pushed how hard you want IMO.

[hydrusnetwork]

Hey, sorry about the late reply here. I've been a little short on time.

Your client is running unusually slow, so I suspect your hard drive or CPU is busy doing something else. Defraggers can slow hydrus down a lot when they do a defrag, or if your hard drive has very high latency (for instance if you do no defragging at all). How much spare space is there on your drive? Is it an HDD or an SSD? Is it very old, or connected to your computer over an unusual USB setup for instance?

Searching for a very large number of files can also slow things down, so try not to view more than, say, 10,000 files into view at once.

I store about 275,000 files on my laptop client and it generally runs ok. A couple of seconds for most searches that give 500-1000 results.

[CuddleBear92]

yeah, i did figure out why things booted so slowly and such. its connected to the tabs i have open and so on. aswell as when i import and download at the same time and so on.

i have managed to work with this more since i know more about the limits of the program, using it every day now to import the huge collection i already have and so on.

only performance gripe i have now is that the UI freezes alot when doing more than two things at one point, i think its limited on the CPU power it can pull or something. i am running this on an HDD and not my SSD's as i cannot change the location of my folders it seems. would love to be able to have the database and program on one drive and have all the files in the collection on a larger hard drive as its getting close to a terra ATM and i would like to use this for most of my things in my H Collection.

[hydrusnetwork]

I am glad it is working a bit better. I plan to add multi-drive database storage to the client in the near future.

Unfortunately, I can't do much about the multi-tasking problem though--the language I write Hydrus in, python, can only work on one CPU core at a time. Lots of people have asked the python developers to add multi-core support, but they haven't done it yet. This is why I have to hide most of my high-CPU processing jobs in idle time, so as not to completely freeze the gui. If this stuff proves a big problem in the future, I can look into putting certain jobs into separate processes with a library called multiprocessing, although that is an ugly solution.

Please keep me updated on how you get on with your large collection.

[CuddleBear92]

yeah, once i figured out that the program has issues with more than two things are happening at one point it went pretty smoothly. im still importing and still downloading tags tho as the local files does not contain tags (or atleast not all of them) im only at 70k files ATM but it will be closer to a mill when im done im sure, still alot to do with it. but it being slow on imports in general just locks up the program to do other stuff in general sadly. if i could import one than one folder at a time or import and download or even download from two tabs at one point it would go alot smoother and faster IMO.

something like a shared multi download tab for boorus would help alot too IMO as it would only have one download queue and not multiple all the time also cutting the amount its doing.... then again, it prob would download on all sites at the same time tho.

one thing that could be done is having a processing queue that tracks everything that gets added to the program and then finalizing the import on idle time and so on. but i guess it could have some issues with doing it that way aswell, i just look at how JMM does it and it queues up all its commands and workload on different queues depending on what it does then works them down instead of waiting for things to save between each thing the program does.

EDIT: final thoughts, the main issue with doing anything with hydrus is that the ui locks up, i cant navigate the menus. i can edit tags or delete images i dont want on the fly of an import/download. i think the process needs to be changed someway for it to actually work on the fly while using the program. something like a work queue will prob help. i did ask the jmm dev if he could think about something that might help hydrus as he.

as for the separate process, i for one dont care if its ugly aslong as it works and dont loose my files or database im happy, rather have speed and functionality over the hole ui locking up when doing anything other than just browsing files.

[CuddleBear92]

just talked to the jmm dev about tech info and that jazz. he said that multi-treading would still not be a problem with python even with a single core used as they can still multi-tread.

also he said that the main issue is the Sql Lite in general and gave me a few links. SQLLite locks up its database when accessing it and that is the base issue it seems. look at the concurrency section on this page (with the tooltip): http://db-engines.com/en/system/Firebird%3BSQLite

he also said that you should read this page as it has alot of the issues hydrus has ATM: http://stackoverflow.com/questions/2863595/sqlite-firebird-does-any-of-them-support-multiple-concurrent-write-access

performance wise sqlite should be avoided whenever possible and in terms of other performance changes you might need to make later down the line then switching database type would be the simplest.

EDIT: final notes that i forgot to write in the original comment, you should switch to Firebird as a Database. he also said that jmm also had alot of the same issues when using sqlite as a database.

@hydrusnetwork

[misakitchi]

I just started to use your soft thanks to AreFin! :) I love it! ^_^

But i have the same problem of GUI freeze... :( Can you do something to go to MSSQL? or its too hard? Or can you give me some advice/tricks?

Thx for this soft! :)

PS: i just create a new server for Hydrus on Discord if you want join :) https://discord.gg/0glC2RfNXebiByBz

[ScoreUnder]

I don't think MSSQL would ever be an option for Hydrus. Things like MSSQL, MySQL, PostgreSQL, etc, all require some heavyweight software running in the background. Firebird and RavenDB sound reasonable but I think it would be wise to first see how much you can squeeze out of SQLite. Then regarding GUI freezes (I don't think I've seen any on my end - only a few hiccups when first rendering the list), it would be best to ensure that any potentially slow operation takes place on another thread so that it doesn't block the GUI. After that I think it would be wiser to consider other database types. At this stage I think switching would be a premature optimization.

[CuddleBear92]

Hey, issues here have been reenabled and will be used for bug reports/feature requests in the future.

We are currently clearing out all the existing issues from 2017 and prior. If your bug report or feature request is still relevant, please create a new issue (or use one of the other channels; see https://hydrusnetwork.github.io/hydrus/).

[CuddleBear92]

Staying closed, this is dumb. Don't need another database. https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md