Database locking error

[psychoflame]

First off i gotta say this is an excellent piece of software. Now i have been having problems with Mylar and the database locking. The problem used to be i would leave Mylar running continuously for a couple days and then the database locking error would come up. A quick restart would fix the problem until the database locked again. However after these recent commits within the last month or so the problems become much more frequent. Mylar locks up every day or so. Here are the logs

ing to avoid hammering 12-Apr-2014 04:35:03 - INFO :: Thread-23 : More than one search provider given - trying next one. 12-Apr-2014 04:35:03 - INFO :: Thread-23 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 9 (2007) using experimental 12-Apr-2014 04:35:05 - INFO :: Thread-23 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using usenet-crawler (newznab)

12-Apr-2014 04:35:05 - INFO :: Thread-23 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 04:36:06 - INFO :: Thread-23 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 04:37:06 - INFO :: Thread-23 : More than one search provider given - trying next one. 12-Apr-2014 04:37:06 - INFO :: Thread-23 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using experimental 12-Apr-2014 04:37:07 - INFO :: Thread-23 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using usenet-crawler (newznab) 12-Apr-2014 04:37:08 - INFO :: Thread-23 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 04:38:08 - INFO :: Thread-23 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 04:39:09 - INFO :: Thread-23 : More than one search provider given - trying next one. 12-Apr-2014 04:39:09 - INFO :: Thread-23 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using experimental 12-Apr-2014 10:28:56 - INFO :: Thread-25 : Initiating NZB Search scan at request ed interval of 360 minutes. 12-Apr-2014 10:28:56 - INFO :: Thread-26 : Retrieving latest version information from github 12-Apr-2014 10:28:56 - INFO :: Thread-26 : Comparing currently installed version with latest github version 12-Apr-2014 10:28:56 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using usenet-crawler (newznab) 12-Apr-2014 10:28:56 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:28:56 - INFO :: Thread-26 : Mylar is up to date 12-Apr-2014 10:29:57 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:30:58 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:31:59 - INFO :: Thread-25 : More than one search provider given - trying next one. 12-Apr-2014 10:31:59 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using experimental 12-Apr-2014 10:32:03 - INFO :: Thread-25 : Couldn't find Issue 6 of Zombies vs C heerleaders(2012). Status kept as wanted. 12-Apr-2014 10:32:03 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 9 (2007) using usenet-crawler (newznab) 12-Apr-2014 10:32:03 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:33:04 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:34:05 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:35:05 - INFO :: Thread-25 : More than one search provider given - trying next one. 12-Apr-2014 10:35:05 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 9 (2007) using experimental 12-Apr-2014 10:35:07 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using usenet-crawler (newznab)

12-Apr-2014 10:35:07 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:36:08 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:37:09 - INFO :: Thread-25 : More than one search provider given - trying next one. 12-Apr-2014 10:37:10 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using experimental 12-Apr-2014 10:37:11 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using usenet-crawler (newznab) 12-Apr-2014 10:37:11 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:38:12 - INFO :: Thread-25 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 10:39:13 - INFO :: Thread-25 : More than one search provider given - trying next one. 12-Apr-2014 10:39:13 - INFO :: Thread-25 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using experimental 12-Apr-2014 16:28:56 - INFO :: Thread-27 : Initiating NZB Search scan at request ed interval of 360 minutes. 12-Apr-2014 16:28:56 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using usenet-crawler (newznab) 12-Apr-2014 16:28:56 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:28:56 - INFO :: Thread-28 : Retrieving latest version information from github 12-Apr-2014 16:28:56 - INFO :: Thread-28 : Comparing currently installed version with latest github version 12-Apr-2014 16:28:56 - INFO :: Thread-28 : Mylar is up to date 12-Apr-2014 16:29:57 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:30:58 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:31:58 - INFO :: Thread-27 : More than one search provider given - trying next one. 12-Apr-2014 16:31:58 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using experimental 12-Apr-2014 16:32:01 - INFO :: Thread-27 : Couldn't find Issue 6 of Zombies vs C heerleaders(2012). Status kept as wanted. 12-Apr-2014 16:32:01 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 9 (2007) using usenet-crawler (newznab) 12-Apr-2014 16:32:01 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:33:02 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:34:02 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:35:07 - INFO :: Thread-27 : More than one search provider given - trying next one. 12-Apr-2014 16:35:07 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 9 (2007) using experimental 12-Apr-2014 16:35:09 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using usenet-crawler (newznab)

12-Apr-2014 16:35:09 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:36:10 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:37:11 - INFO :: Thread-27 : More than one search provider given - trying next one. 12-Apr-2014 16:37:11 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Star Wars: Clone Wars Adventures issue: 10 (2008) using experimental 12-Apr-2014 16:37:12 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using usenet-crawler (newznab) 12-Apr-2014 16:37:12 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:38:13 - INFO :: Thread-27 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 16:39:14 - INFO :: Thread-27 : More than one search provider given - trying next one. 12-Apr-2014 16:39:14 - INFO :: Thread-27 : Shhh be very quiet...I'm looking for Futurama Comics issue: 70 (2014) using experimental 12-Apr-2014 22:28:56 - INFO :: Thread-30 : Initiating NZB Search scan at request ed interval of 360 minutes. 12-Apr-2014 22:28:56 - INFO :: Thread-32 : Retrieving latest version information from github 12-Apr-2014 22:28:56 - INFO :: Thread-29 : Starting update for 163 active comics

12-Apr-2014 22:28:56 - INFO :: Thread-31 : Weekly pull list present - checking i f it's up-to-date.. 12-Apr-2014 22:28:56 - INFO :: Thread-32 : Comparing currently installed version with latest github version 12-Apr-2014 22:28:56 - INFO :: Thread-30 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using usenet-crawler (newznab) 12-Apr-2014 22:28:56 - INFO :: Thread-30 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 22:28:56 - INFO :: Thread-29 : Found comic directory: D:\Comics\Star Wars Legacy (2013) 12-Apr-2014 22:28:56 - INFO :: Thread-32 : Mylar is up to date 12-Apr-2014 22:28:56 - INFO :: Thread-31 : No new pull-list available - will re- check again in 24 hours. 12-Apr-2014 22:28:56 - INFO :: Thread-31 : Checking the Weekly Releases list for comics I'm watching... 12-Apr-2014 22:29:18 - WARNING :: Thread-29 : Database Error: database is locked

12-Apr-2014 22:29:18 - WARNING :: Thread-29 : sqlresult: UPDATE comics SET Statu s = ? WHERE ComicID = ? 12-Apr-2014 22:29:41 - WARNING :: Thread-29 : Database Error: database is locked

12-Apr-2014 22:29:41 - WARNING :: Thread-29 : sqlresult: UPDATE comics SET Statu s = ? WHERE ComicID = ? 12-Apr-2014 22:29:57 - INFO :: Thread-30 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 22:30:04 - WARNING :: Thread-29 : Database Error: database is locked

12-Apr-2014 22:30:04 - WARNING :: Thread-29 : sqlresult: UPDATE comics SET Statu s = ? WHERE ComicID = ? 12-Apr-2014 22:30:27 - WARNING :: Thread-29 : Database Error: database is locked

12-Apr-2014 22:30:27 - WARNING :: Thread-29 : sqlresult: UPDATE comics SET Statu s = ? WHERE ComicID = ? 12-Apr-2014 22:30:50 - WARNING :: Thread-29 : Database Error: database is locked

12-Apr-2014 22:30:50 - WARNING :: Thread-29 : sqlresult: UPDATE comics SET Statu s = ? WHERE ComicID = ? 12-Apr-2014 22:30:58 - INFO :: Thread-30 : pausing for 60 seconds before continu ing to avoid hammering 12-Apr-2014 22:31:23 - WARNING :: Thread-31 : Database Error: database is locked

12-Apr-2014 22:31:23 - WARNING :: Thread-31 : sqlresult: SELECT * FROM comics WH ERE ComicID=? 12-Apr-2014 22:31:56 - WARNING :: Thread-31 : Database Error: database is locked

12-Apr-2014 22:31:56 - WARNING :: Thread-31 : sqlresult: SELECT * FROM comics WH ERE ComicID=? 12-Apr-2014 22:31:59 - INFO :: Thread-30 : More than one search provider given - trying next one. 12-Apr-2014 22:31:59 - INFO :: Thread-30 : Shhh be very quiet...I'm looking for Zombies vs Cheerleaders issue: 6 (2012) using experimental 12-Apr-2014 22:32:01 - INFO :: Thread-30 : Couldn't find Issue 6 of Zombies vs C heerleaders(2012). Status kept as wanted. 12-Apr-2014 22:32:29 - WARNING :: Thread-31 : Database Error: database is locked

12-Apr-2014 22:32:29 - WARNING :: Thread-31 : sqlresult: SELECT * FROM comics WH ERE ComicID=? 12-Apr-2014 22:33:02 - WARNING :: Thread-31 : Database Error: database is locked

12-Apr-2014 22:33:02 - WARNING :: Thread-31 : sqlresult: SELECT * FROM comics WH ERE ComicID=? 12-Apr-2014 22:33:35 - WARNING :: Thread-31 : Database Error: database is locked

12-Apr-2014 22:33:35 - WARNING :: Thread-31 : sqlresult: SELECT * FROM comics WH ERE ComicID=? 12-Apr-2014 22:34:07 - WARNING :: Thread-30 : Database Error: database is locked

12-Apr-2014 22:34:07 - WARNING :: Thread-30 : sqlresult: SELECT * from comics WH ERE ComicID=? AND ComicName != 'None' 12-Apr-2014 22:34:40 - WARNING :: Thread-30 : Database Error: database is locked

12-Apr-2014 22:34:40 - WARNING :: Thread-30 : sqlresult: SELECT * from comics WH ERE ComicID=? AND ComicName != 'None' 12-Apr-2014 22:35:13 - WARNING :: Thread-30 : Database Error: database is locked

12-Apr-2014 22:35:13 - WARNING :: Thread-30 : sqlresult: SELECT * from comics WH ERE ComicID=? AND ComicName != 'None' 12-Apr-2014 22:35:46 - WARNING :: Thread-30 : Database Error: database is locked

12-Apr-2014 22:35:46 - WARNING :: Thread-30 : sqlresult: SELECT * from comics WH ERE ComicID=? AND ComicName != 'None' 12-Apr-2014 22:36:19 - WARNING :: Thread-30 : Database Error: database is locked

12-Apr-2014 22:36:19 - WARNING :: Thread-30 : sqlresult: SELECT * from comics WH ERE ComicID=? AND ComicName != 'None' 13-Apr-2014 04:28:56 - INFO :: Thread-33 : Initiating NZB Search scan at request ed interval of 360 minutes. 13-Apr-2014 04:28:56 - INFO :: Thread-34 : Retrieving latest version information from github 13-Apr-2014 04:28:56 - INFO :: Thread-34 : Comparing currently installed version with latest github version 13-Apr-2014 04:28:56 - INFO :: Thread-34 : Mylar is up to date 13-Apr-2014 04:29:28 - WARNING :: Thread-33 : Database Error: database is locked

13-Apr-2014 04:29:28 - WARNING :: Thread-33 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 04:30:00 - WARNING :: Thread-33 : Database Error: database is locked

13-Apr-2014 04:30:00 - WARNING :: Thread-33 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 04:30:33 - WARNING :: Thread-33 : Database Error: database is locked

13-Apr-2014 04:30:33 - WARNING :: Thread-33 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 04:31:06 - WARNING :: Thread-33 : Database Error: database is locked

13-Apr-2014 04:31:06 - WARNING :: Thread-33 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 04:31:39 - WARNING :: Thread-33 : Database Error: database is locked

13-Apr-2014 04:31:39 - WARNING :: Thread-33 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 10:28:56 - INFO :: Thread-35 : Initiating NZB Search scan at request ed interval of 360 minutes. 13-Apr-2014 10:28:56 - INFO :: Thread-36 : Retrieving latest version information from github 13-Apr-2014 10:28:56 - INFO :: Thread-36 : Comparing currently installed version with latest github version 13-Apr-2014 10:28:56 - INFO :: Thread-36 : Mylar is up to date 13-Apr-2014 10:29:28 - WARNING :: Thread-35 : Database Error: database is locked

13-Apr-2014 10:29:28 - WARNING :: Thread-35 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 10:30:00 - WARNING :: Thread-35 : Database Error: database is locked

13-Apr-2014 10:30:00 - WARNING :: Thread-35 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 10:30:33 - WARNING :: Thread-35 : Database Error: database is locked

13-Apr-2014 10:30:33 - WARNING :: Thread-35 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 10:31:06 - WARNING :: Thread-35 : Database Error: database is locked

13-Apr-2014 10:31:06 - WARNING :: Thread-35 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 10:31:39 - WARNING :: Thread-35 : Database Error: database is locked

13-Apr-2014 10:31:39 - WARNING :: Thread-35 : sqlresult: SELECT * from issues WH ERE Status="Wanted" 13-Apr-2014 12:31:56 - WARNING :: CP Server Thread-4 : Database Error: database is locked 13-Apr-2014 12:31:56 - WARNING :: CP Server Thread-4 : sqlresult: SELECT * from comics order by ComicSortName COLLATE NOCASE 13-Apr-2014 12:32:29 - WARNING :: CP Server Thread-4 : Database Error: database is locked 13-Apr-2014 12:32:29 - WARNING :: CP Server Thread-4 : sqlresult: SELECT * from comics order by ComicSortName COLLATE NOCASE 13-Apr-2014 12:33:02 - WARNING :: CP Server Thread-4 : Database Error: database is locked 13-Apr-2014 12:33:02 - WARNING :: CP Server Thread-4 : sqlresult: SELECT * from comics order by ComicSortName COLLATE NOCASE 13-Apr-2014 12:33:35 - WARNING :: CP Server Thread-4 : Database Error: database is locked 13-Apr-2014 12:33:35 - WARNING :: CP Server Thread-4 : sqlresult: SELECT * from comics order by ComicSortName COLLATE NOCASE 13-Apr-2014 12:34:08 - WARNING :: CP Server Thread-4 : Database Error: database is locked 13-Apr-2014 12:34:09 - WARNING :: CP Server Thread-4 : sqlresult: SELECT * from comics order by ComicSortName COLLATE NOCASE

[evilhero]

I've come across this myself, it's due to having too many processes attempt to update (write) to the db, and sqlite3 only allows for one write process at any given time. Sqlite3 locks the entire table, whereas other db programs (sql for instance) just locks the row, which allows for multiple writes to occur on the same table simultaneously.

I've looked into having just one thread dedicated to writing, any number dedicated to reading (cause it doesn't affect anything), and then have a 'worker queue' where things get queued if the writing process is busy - but it's a pretty complicated process thus far (at least for myself). I am looking into this, and I'd rather just not jump right to sql (cause then there'd be a crapload of backend stuff to do and sqlite3 has a smaller memory footprint from what I can gather and sql may be abit of overkill for this kind of program), but if it turns out that's the only recourse I might have no choice in order to get rid of the persistent error.

[bmfrosty]

Absolutely loving this, and glad that I'm not uniquely having this same problem.

You may also be able to get away with a lock file of some sort for this. Just enough to prevent contention for the database.

[evilhero]

Well there is a lock already present when it tries to access the db - and I've looked at other ways to try and get it to lock/queue up but nothing has worked thus far. It always seems to get the database locking, when two scheduled jobs are having contention issues.

I figured out that it's partially due to the scheduler that Mylar uses to schedule the jobs and they tend to conflict - I've also noticed that I haven't seen these errors occur in sickbeard (headphones yes, which is where the problem inherited from). So in light of that, on my test box - I'm using the scheduler that sickbeard uses but I've had to re-do a bit of every module in order to accommodate the inclusion of Classes instead of just having functions. I'm going to let it run for me for a bit to see if I can get it to lock, but failing that I'll probably push it out to it's own branch before flipping it into the development branch (this 'new' branch will include a new logging system, the new scheduling, and being able to select the searching order for search providers).

[apowell656]

For Windows users I have temporarily added a task using Windows Task Scheduler.Triggers are startup and start program at 4:00 am, with the task not running for more than one day, until there is a fix.

[pedrovanzella]

Would it be too hard to add the option to use something like postgresql? That would fix the locks nicely.

[evilhero]

Definitely possible, especially with a decent database adaptor to python. I just don't want to alienate those users that haven't experienced the db locks and make the installation process much more complicated with having to setup postgresql.

But since all the db calls go through the same module, I think it could be feasible to have an option to use either sqlite3 or postgresql db.

[bmfrosty]

If you're going in that direction may as well open up mysql as well. I'm in the process off opening up MySQL for XBMC anyway.

On Mon, Nov 3, 2014 at 9:12 AM, evilhero notifications@github.com wrote:

Definitely possible, especially with a decent database adaptor to python. I just don't want to alienate those users that haven't experienced the db locks and make the installation process much more complicated with having to setup postgresql.

But since all the db calls go through the same module, I think it could be feasible to have an option to use either sqlite3 or postgresql db.

— Reply to this email directly or view it on GitHub https://github.com/evilhero/mylar/issues/667#issuecomment-61512023.

[pedrovanzella]

From a usability standpoint, it's easy to solve the problem. Just have sqlite as default, and add an option to switch to postgres or mysql in the config page. Add an export / import script and everyone is happy.

[evilhero]

Even with the import / export (which is a good idea in itself), it will still require a fairly substantial rework of the existing code to accommodate the different db's. Definitely something I'm looking into though, as pretty much every db select statement would have to point to the new db calling methods.

[dilbaggins]

i second this emotion. this makes mass import nigh impossible. or if unattended i get a db lock on a weekly. so i have a cron setup to restart automatically every day at midnight. perhaps as you said @evilhero some type of governor that lets one write in at a time or a move to MySQL.

[evilhero]

I've modified alot of the way that Mylar accesses the db in the most recent development commit (4fad909380198715e391f8ce8ce2fa478e03d750) - not so much in the actual coding of it, but moreso in the methodology.

For instance, I found 2 separate instances of external db connections (outside of using the db.py module with Mylar) that connect to the mylar.db and do quite a few writes/reads - the end result if any of these processes were running at a time when Mylar was trying to access the db via the db module it would cause record locks. I've since recoded the Weekly Pull-list and the Import Directory sections (the 2 sections of code I found that had these external processes) - and reverted back to the use of a new scheduler that I had pushed out last year, but the cpu usage was too high for it be considered for use at that time (I figured out since why, and took out the portion that was causing the cpu spiking).

The result: under the latest development commit, I have simultaneously ran:

• a manual post-processing of 37 issues
• a re-creation of the weekly pull list (Recreate pull-list)
• a db update for all active comics
• an import of a directory containing several series that were/were not on the watchlist.
• added a series.

No locks. No locks. No locks. No locks. No locks.

I'm not saying it's solved, but I'm feeling it's avoiding the locking alot better and is a definite step in the right direction as I haven't had any locks in the last few days (since I finished the code).

I have some other modules I want to clean up and make sure they follow the same workflow so there's no stragglers, so that's coming in the pipe over the next few commits most likely.

Once I finish those, then depending on how things are going, I'll think about making a different branch for MySQL/Postgresql since having it all in one branch at this point is really confusing.

[ghost]

I want to love this app, but not being able to do a mass import is really a no starter for me. I've got about 2500 issues in separate folders and I get the locking issue nearly every time if I try to import more than a single series. I'm going to keep playing with it, but I'm not looking forward to importing all 170 series one by one.

[evilhero]

Are you on the development or master branch of mylar? Development has the fixes that should address the database locking problems.

[ghost]

Yeah, apologies. I was just reading about a similar issue in the forums. I'll switch to the dev branch and give it a go.

[evilhero]

No worries, and please if you could give an update - I haven't had any responses /issues related to the new fixes thus far so aside from working for myself, I don't know how they're working for others yet

[ghost]

It definitely works better; I was able to do a mass import of two series that only had 4 or 5 issues each. But I tried 10 series (total of about 25 issues) and it locked up again. Let me know if you'd like logs and I'll post them when it's up and running again...seems like I have to close the command prompt as the restart command doesn't seem to do much.

[evilhero]

Yeah if you could, I'd need to see what was trying to access the db concurrently which would have caused the locking. I had fixed a few problematic areas and knew there might be others but hadn't come across any during my testing period.

When you were selecting the 10 series, were you using the checkbox/ drop down menu or another method to start the import process?

[ghost]

I was using the checkbox to select, then using the dropdown to import.

One other thing that may have contributed; previously I was importing to the same directory. After reading that forum post, I started importing to a new, empty dir.

[evilhero]

Well importing to the same directory would definitely cause problems as it's trying to write files to the directory it's trying to import from while updating both what's been done and what has finished.

I'll have to see about adding in some checks so that might not be allowed in future iterations ;)

Let me know how the import goes when you're importing from a different directory (or to a new one).. Hopefully it will fair abit better.

[ghost]

I selected 6 series with a total of 163 issues using the check boxes, then start import from the dropdown. Looks like it made it a ways, then the DB locked up. Here are the logs starting at the import

[ghost]

I'm also going to try importing from a dir that has just a few series in it rather than the entire 170 or so.

EDIT: Yeah, importing in smaller batches definitely helps. I was able to complete the above batch import with no problems.

[evilhero]

Can I ask what the batch sizes roughly were? I'll be looking into this later tonight, so I can try to go beyond whatever threshold you had done so I can test for locking..

[ghost]

On the last one I tried 5 series with approx 160 issues.

[lodasi]

I'm experiencing this error now on development branch. It got to be to the point where I just backed up my old mylar.db and started anew, but the error has popped up again. Was there an update to any DB calls in this weeks commits?

[evilhero]

If it's happening on shortly after startup them odds are it's the helpers call to update issue dates in 02fa5baf1e3b8516060edabb65142af8ebb1afe5. I'm guessing that something is firing off (Rss, search on startup, etc) while it's updating and locking up the dB due to it.

I was gonna drop that commit, but haven't as of yet. Can you confirm that it's happening after a restart (even a few mins after)

[lodasi]

Yes, it was happening right after restart. Unfortunately, the end result was that it was never unlocking itself and it became impossible to do anything as the web interface service would never start. I'm not seeing these errors as much, but it made me concerned that the database was corrupted, hence the back up and start anew.

It does occur when trying to add a new series outside of the Manage Import. sqlresult: UPDATE comics SET Type = ?, Status = ?, ComicName_Filesafe = ?, ComicName = ?, ComicImageURL = ?, ComicPublished = ?, ComicImageALTURL = ?, DynamicComicName = ?, ComicVersion = ?, ComicLocation = ?, DetailURL = ?, ComicPublisher = ?, ComicYear = ?, ComicSortName = ?, ComicImage = ?, DateAdded = ?, Total = ? WHERE ComicID = ?

[evilhero]

Yeah, it was probably the initial db lock with the new helpers function I had added in order to clean up bad issue table entries that have an impact on the weekly pull's working properly. I'm going to guess that the error that's being thrown is because the one that has the actual lock on the file won't release it and then it just continually queues.

I'll push out a commit right now to remove that call on startup so that the locks shouldn't happen again (it's been a very long time since the locking error was reported), and I'll try to figure out a better method to implement the issue updating problem.

[lodasi]

The DB locking is still occurring. It takes around 3-4 minutes for the web interface to initialize from a cold start.
Redacted log: http://pastebin.com/Sm5jWjUB

[evilhero]

Make sure you delete the init.pyc file in the mylar subdirectory with mylar is not running. Then start mylar with the -v mode for verbose logging.

The only other thing is to turn off the search on start method as that has tended to cause problems in the past (although if you had it enabled previously when there was no problems, it might not have mattered)

[lodasi]

Yeah, I have not updated my configuration in months, so this is a new issue. Tried deleting the init.pyc file, but this didn't help either. I will try with the nzb search on start disabled.

Edit: Still get a database lock error in WEEKLYCHECK.

[evilhero]

Honestly that should have done it as there's nothing else this last week that would have affected it to this degree, and you're the only one whose brought it forth thus far.

Can you start mylar with the -v option via command line and then paste the log again? I need to see more info as to the threads and what's being run when and how, because atm it seems to be isolated to just yourself for some reason.

[lodasi]

http://pastebin.com/YSutHbcb

[jimbojnr81]

Im having the exact same issues as lodasi. I'm on the latest Dev. I've tried the steps above, turning off search at startup, delete init.pyc and still get the lock error. The error happens when i try to load any page other than the logs and the initial manage page but not when i try to go to manage issues or manage comics. I have cleared my wanted list just so there are no searches at all. I don't really want to start from scratch as i have a lot of older series that are incomplete and it just took way too long to add them all to mylar to begin with.

Anything i can give you to help, let me know.

If it helps i was also getting the failed files error that lodasi is getting at line 226 and, for me, it was because they didn't have an issue number in them.

[evilhero]

I can't dupe this for the life of me... Is there anyway you guys or one can send me your dB file and your ini (just remove the api keys), then I can test and see what's happening in RealTime.

What os are you guys running?

[jimbojnr81]

I'm on windows, I've just installed a fresh copy of mylar alongside my other one, copied the DB across but just entered in the config data manually and so far it seems to be working. I'm doing all the same actions i was trying to do with the original install and all is working, so far.. Ill let you know if it goes pair shaped..

[jimbojnr81]

windows 10 64bit

[jimbojnr81]

Spoke to soon.. separate error to this thread but i started getting this one not long before the database lock error

WindowsError: [Error 32] The process cannot access the file because it is being used by another process

[lodasi]

I'm also on Windows 10 x64.

[jimbojnr81]

ok other than the above error, everything else seems to be working ok. I've been able to grab an issue, post process, and even manually meta tag some other issues that i was having trouble with before and no dbase lock.

[jimbojnr81]

something must have gone a bit off on my previous install.

[evilhero]

The windows error 32,does it mention something about the logging file and not being able to roll over? The locked log file is a know problem with windows machines, but if you configure the concurrent log handler within mylar, you won't get those lock messages again (if that's what it's indicating).

I have a win10x64 here I can boot up, I'll try throwing stuff at it and seeing if I can duplicate it on there.

[lodasi]

From memory, the logging issue typically throws a traceback repeatedly when the log file reaches the max size and doesn't flip to the new file. This is a much different error.

[evilhero]

Yes but the actual traceback is the error 32 due to windows locking the log file and won't release it to spawn a new log file. It's a graceful error as mylar continues, but it stops writing to the log file due to it.

@lodasi: it might be worth doing what @jimbojnr81 did - copy the dB file and .ini file to another location, trash your existing mylar dir, and reinstall a clean copy. Then move back the dB and hand code in the ini. You may also want to back up your cache dir as that holds all the jpgs for the covers for the series, and if you don't the pages would show image placeholders until you refresh the series.

I tried on my win10x64 machine using 3 diff local mylar installs and no dB locks on any of them.

[jimbojnr81]

Yeah the fresh install with transferring the DB across seems to have done the trick, it's been a couple of hours now and no DB lock, also no error 32 as well but that could just come up when booting mylar up.

[lodasi]

Fresh install from zip worked. Ran once to generate the ini and db and then shutdown/replaced the db and edited the ini.

Restarted and everything worked well. Noticed a lot of journaling activity probably due to the WEEKLYCHECK thread.

Butl I updated because it said I was 16 commits behind.

Then the error returned after the update.

http://pastebin.com/28jYXCGu

[evilhero]

Weird, there's something amiss somewhere. You don't have mylar running as a daemon or something in the background do you?

The only other thing I can suggest is to send the dB and apikey-less config.ini, and I can sideload on the machines that I have in an attempt to isolate and figure it out.

What version of python are you running mylar against? You're not running mylar via a docker or the like are you?

[lodasi]

Please let me know how I can share the link with just you.

[evilhero]

Once you email, I'll remove my addy so them spamerz don't get me

[lodasi]

Done! Thanks!

[jimbojnr81]

Just an FYI for me it's still going strong. I just restarted Mylar and it seems the error 32 has gone now too.

hope you work it out lodasi, anything i can do to help let me know!