[BUG]UI becomes unresponsive, locks up

[appleb4]

Describe the bug After some time (at least 30 minutes), the UI becomes completely unresponsive. I can't drag the window, click any buttons, and right clicking the system tray icon will bring up the context menu, but I'll be unable to close it or click on anything. Trying to open the program by clicking the taskbar icon will play the windows beep sound as if something is holding focus from it

To Reproduce Steps to reproduce the behavior:

1. Profile URL: Any profile, I've tried all combinations of profiles I follow, which are only on twitter. I've deleted them all to try single profiles, the same thing happens
2. I have the autodownloader set to 10 minute intervals

Log data I'm not able to find it in the logs folder, but the only entry from the tab I saw was it trying to connect to downdetector.co.uk

Release information (please complete the following information): OS Windows10 Architecture x64 Version 2022.12.27.0 NET.Framework version 4.8.4084.0

Additional context I think this issue only started happening after version 2022.11.16.0. I've already tried clearing my config files and re-downloading the program. I didn't report this earlier because I wanted to find out how to reproduce it properly, but I don't know what the cause is other than having the program running in the background for awhile.

[AAndyProgram]

It's very strange what you described. I use the SCrawler almost every day. I also use autodownloader. So, I start the Scrawler and close it to tray. When it notifies me that something has been downloaded, I only open the feed to see the new content. I have never seen the SCrawler freeze, except in rare cases when trying to play a video in the feed.

Maybe some message box was popping up in the background? Try switching the app with Alt+TAB and return to the SCrawler in the same way. If other operations are still going on and only the SCrawler is not responding, I bet the problem is in the message box.

In other cases, unfortunately, I need more information, including LOG. The log files are placed in the SCrawler directory in the LOGs folder. Also, please write the profiles of which sites you are trying to download.

[appleb4]

It's very strange what you described. I use the SCrawler almost every day. I also use autodownloader. So, I start the Scrawler and close it to tray. When it notifies me that something has been downloaded, I only open the feed to see the new content. I have never seen the SCrawler freeze, except in rare cases when trying to play a video in the feed.

Maybe some message box was popping up in the background? Try switching the app with Alt+TAB and return to the SCrawler in the same way. If other operations are still going on and only the SCrawler is not responding, I bet the problem is in the message box.

In other cases, unfortunately, I need more information, including LOG. The log files are placed in the SCrawler directory in the LOGs folder. Also, please write the profiles of which sites you are trying to download.

Yeah it's really strange, and on the topic of notifications, i do have them enabled but I've never actually seen one. Where are they supposed to pop up? My log file is completely empty. I think the program being frozen means it's not logging or downloading anything and I need to end the task. I also have no log file in my logs folder I'm following a single profile, which at this moment is downloading fine and has been since I made this post. https://twitter.com/We_Like_Dogs

I've tried alt tabbing, along with windows key+D to minimize everything, I don't have any background popups.

In the meantime I'm going to try a completely new install in a new location, and new cookies to see what happens

[AAndyProgram]

Where are they supposed to pop up?

In Windows notifications. Unless you disable them in SCrawler settings, they should appear when autodownloader has downloaded something and/or when you download profiles manually and that task is complete.

I'm following a single profile, which at this moment is downloading fine and has been since I made this post.

So is everything okay now?

[appleb4]

Where are they supposed to pop up?

In Windows notifications. Unless you disable them in SCrawler settings, they should appear when autodownloader has downloaded something and/or when you download profiles manually and that task is complete.

I'm following a single profile, which at this moment is downloading fine and has been since I made this post.

So is everything okay now?

Thanks, my windows notifications were off. I see notifications now

Not okay yet, it's just hard for me to get a timeframe for when it happens, could be 30 minutes, could be hours. I'm beginning to wonder if it has something to do with fullscreen apps, cause I usually notice it after I close a game though that could just be coincidence. I was able to see it freeze yesterday by switching windows HDR mode on and off but again that could be unrelated. I'm going to continue running this new install with default settings on another drive to see if it changes anything

[AAndyProgram]

I see notifications now

Cool! 😊

I'm beginning to wonder if it has something to do with fullscreen apps, cause I usually notice it after I close a game though that could just be coincidence.

I really doubt. The SCrawler doesn't look at other processes and doesn't interfere with them. But I think that if your game takes up most of your computer's resources, then I guess this can happen.

I was able to see it freeze yesterday by switching windows HDR mode on and off but again that could be unrelated.

Didn't use it.

I'm going to continue running this new install with default settings on another drive to see if it changes anything

Okay, keep me posted 😉

[appleb4]

Sorry for the delay, I wanted to at least have something to show. It's still happening unfortunately, but I've been able to reproduce it on my end. I've narrowed it down to autodownloader, and maybe an exclusive fullscreen app (still not 100% sure on this because I thought I've seen it happen before while my PC was inactive). I attached a video of it running an autodownloader with 1 minute intervals. I start a game that runs exclusive fullscreen and let the autodownloader run in the background. After a minute, I minimize and the scrawler UI is unresponsive https://user-images.githubusercontent.com/116610866/210155384-5962f1e2-dd7f-4862-8fd9-50e03a2b1b71.mp4

I wasn't able to reproduce this in borderless windowed, but I'm still testing. I've had the UI freeze with all 6 of my tests of a fullscreen app. No log file unfortunately, sorry.

It's pretty consistent with the fullscreen app. Testing on the older 10.18.0 version now,. so far it's working

[AAndyProgram]

Are you saying that 2022.10.18.0 doesn't have this bug? If this is true, I can review code changes to this version, but I don't remember any changes that would cause this error.

[AAndyProgram]

Btw, Happy new year 🎄

[appleb4]

Btw, Happy new year 🎄

Happy New Year

Yeah the bug seems to have been introduced in 2022.11.16.0.

[AAndyProgram]

I've already looked at the code changes since 2022.10.18.0 and don't see anything that could explain this behavior. I'll look again, but I'm not sure that I can identify the bug. For now, I can only suggest that you close SCrawler before starting a game.

[appleb4]

Thanks, it's very weird. I also tried many different settings and compatibility modes and all result in the same problem, I don't understand what could cause this

For now I'll continue using the older version, because I only have this running for twitter and it seems to be working fine

[AAndyProgram]

Hello. Any news on this issue? I still don't face the UI becoming unresponsive.

[drag083]

Just chiming in to say I have also started seeing this behaviour. It seems to only lock up after the program has been running for a couple of days, whereas previous releases were rock solid and never locked up.

[AAndyProgram]

WOW! It's so weird....

[AAndyProgram]

@drag083, just works for 2 days and freeze? Are you putting your PC to sleep or are you just working normally? I run SCrawler when my computer starts or wakes up from sleep. It works all day long. Then I close SCrawler and put my computer to sleep. This is how I use SCrawler. And I have never seen such behavior.

[AAndyProgram]

@appleb4, I watched your video again. I see that SCrawler is not responding.

The first thing I noticed was that the LOG form was not open before starting the game. After closing the game, I see two open LOG forms. How so? Or did you just crop the video?

The second thing I noticed is that while your game was running, SCrawler was downloading something through the "Default" scheduler plan. In the plan settings on the video, I saw that notifications are enabled. Is that possible that something has been downloaded and SCrawler sent you notifications? If so, then maybe the problem is in the notifications?

May you unpack SCrawler to a new folder (clean install) and just run it? Don't add any profiles or scheduler plans. Just start SCrawler then try to play your game and then go back to SCrawler. I need to know when and where the error occurs. Text me if SCrawler is not responding. If the problem is gone, add a new profile to this clean SCrawler (it doesn't matter which one). Don't start downloading and repeat all steps. If the problem still doesn't occur, please add the same scheduler plan as in the video you posted. Restart SCrawler and repeat all steps.

[appleb4]

@appleb4, I watched your video again. I see that SCrawler is not responding.

The first thing I noticed was that the LOG form was not open before starting the game. After closing the game, I see two open LOG forms. How so? Or did you just crop the video?

The second thing I noticed is that while your game was running, SCrawler was downloading something through the "Default" scheduler plan. In the plan settings on the video, I saw that notifications are enabled. Is that possible that something has been downloaded and SCrawler sent you notifications? If so, then maybe the problem is in the notifications?

May you unpack SCrawler to a new folder (clean install) and just run it? Don't add any profiles or scheduler plans. Just start SCrawler then try to play your game and then go back to SCrawler. I need to know when and where the error occurs. Text me if SCrawler is not responding. If the problem is gone, add a new profile to this clean SCrawler (it doesn't matter which one). Don't start downloading and repeat all steps. If the problem still doesn't occur, please add the same scheduler plan as in the video you posted. Restart SCrawler and repeat all steps.

You're correct, the video was cut. Between the cut, the game wasn't running in fullscreen, the autodownloader ran without problem. I opened the logs and changed the game to fullscreen, waited for the autodownloader to run and the UI stopped responding.

I did what you asked, new install in a new location. I first ran it without anything set, no problems with fullscreen game. I then added a twitter profile, no autodwnload. I let the game run for a bit, no problems. Then I added the autodownloader with notifications disabled and the problem came back after it tried downloading. I also tried a new plan with twitter downloads disabled, which ran fine. So it seems to be whenever Scrawler tries to download or check for a download (if there's no new posts) from the autodownloader, notifications don't seem to be the cause.

I'll also add that I'm on an Nvidia RTX 3080 GPU, if that's anything related to with drivers seeing as it only seems to happen when a fullscreen app has priority. The only other thing I can think of that seems different between the older versions that work for me, is the tray icon doesn't change to the blue arrow when downloading, but hiding that tray icon didn't change anything. Very weird!

[AAndyProgram]

You're correct, the video was cut. Between the cut, the game wasn't running in fullscreen, the autodownloader ran without problem. I opened the logs and changed the game to fullscreen, waited for the autodownloader to run and the UI stopped responding.

Got it.

Then I added the autodownloader with notifications disabled and the problem came back after it tried downloading.

As far as I understand the problem is in the autodownloader or in the downloader. Please try again with a clean install. Add a profile that contains a lot of download data (eg https://twitter.com/nasahqphoto) that may continue to download when you start playing the game. Don't create a scheduler plan. Before you start playing, please start to download manually and start playing. Then return to SCrawler. This action is intended to detect an error in the autodownloader thread or in the downloader because they run on separate threads.

I also tried a new plan with twitter downloads disabled, which ran fine.

What does it mean? Are you disabled Twitter downloads in the Settings-Twitter or in the scheduler plan settings? Anyway, if you disable Twitter profiles downloading and the only profile in the SCrawler is a Twitter profile, then I think that the problem is in the downloader, not in the autodownloader, because if you disabled Twitter profiles downloading, then there is nothing to download.

The only other thing I can think of that seems different between the older versions that work for me, is the tray icon doesn't change to the blue arrow when downloading, but hiding that tray icon didn't change anything.

This is just a design improvement. I don't think it has any effect. 🤷🏼‍♂️

I'll also add that I'm on an Nvidia RTX 3080 GPU, if that's anything related to with drivers seeing as it only seems to happen when a fullscreen app has priority.

I don't know...

Thank you for your help and time. I appreciate you taking the time to do what I'm asking.

[AAndyProgram]

I highly doubt that the tray icon can affect anything, but if you think so, I created three beta versions.

In this version (SCrawler_no_tray_change_x64.zip), the change of tray icon (to the arrow) is disabled.

In these versions (SCrawler_tray_change_1_x64.zip and SCrawler_tray_change_2_x64.zip), the change of tray icon (to the arrow) is enabled.

Please, try all of them

[appleb4]

For all the time your program saves me, I'm happy to help.

Please try again with a clean install. Add a profile that contains a lot of download data (eg https://twitter.com/nasahqphoto) that may continue to download when you start playing the game. Don't create a scheduler plan. Before you start playing, please start to download manually and start playing. Then return to SCrawler. This action is intended to detect an error in the autodownloader thread or in the downloader because they run on separate threads.

I tried this, it downloaded the profile in the background just fine. Once I added the autodownloader though, it locked up again

What does it mean? Are you disabled Twitter downloads in the Settings-Twitter or in the scheduler plan settings? Anyway, if you disable Twitter profiles downloading and the only profile in the SCrawler is a Twitter profile, then I think that the problem is in the downloader, not in the autodownloader, because if you disabled Twitter profiles downloading, then there is nothing to download.

Yeah I tried excluding twitter downloads through the schedule settings. I wanted to see if the autodownloader running, but not downloading would change anything, but I also have no idea if that would make sense to test, much like the tray icon idea. I'm grasping at straws here 😛

Which leads into the beta versions. All results were the same unfortunately. I ran all of them as fresh installs and using the same steps as before, the UI would lock up on all of them.

In a few days when I have extra time, I'll try to recreate this issue on another cheap PC I have, it has no dedicated GPU.

[AAndyProgram]

I tried this, it downloaded the profile in the background just fine. Once I added the autodownloader though, it locked up again

Okay. So at least now we know where exactly an error occurs.🙂 It remains only to determine what is happening. I'll think about. Thank you very much for your time! 😊

[AAndyProgram]

Since I know where an error occurs, I need to know the exact location in the code. I've created this release (not for work, just for tests): SCrawler_beta_not_for_work.zip. I added a detailed log to the autodownloader. So please, if you have time, run this with the steps that lead to the error. This log will be automatically saved each time new data is added to. The log will be stored in the SCrawler folder: LOGs\PLAN_INTERNAL_LOG.txt. When you do, please submit the log that will be generated.

[appleb4]

PLAN_INTERNAL_LOG.txt Here's the log file I got. I started the program, added a profile, manually downloaded some photos and then set up the autodownloader to download after 1 minute.

[AAndyProgram]

then set up the autodownloader to download after 1 minute.

Have you configured to download every minute? It's too often. Have you configured the same value in your working version?

As I can see, SCrawler has successfully started and finished downloading users according to plan. But after that, a new downloading immediately begins. Via the same plan. And this time it hangs at the moment when the plan sends users to the downloader.

I need time to think. This is so weird. I expected to see this behavior in other functions.

[appleb4]

Have you configured to download every minute? It's too often. Have you configured the same value in your working version?

I only configured it to 1 minute to quickly reproduce the bug, I usually have it set to 10 minutes which works fine in the other version. I also tried matching the values exactly between the versions, I noticed that in v2022.10.18 (the latest one that works for me), the default "delay" setting in the autodownloader is 0, and the newer versions it's defaulted to 1. I tried changing those around, and got the same result. Not that I really expected it to make a difference but I'm just letting you know. Tomorrow I should be able to test on another PC(If I can properly restore it, bad HDD)

[AAndyProgram]

the default "delay" setting in the autodownloader is 0, and the newer versions it's defaulted to 1. I tried changing those around, and got the same result.

It was added to delay the execution of tasks right after SCrawler is launched.

[appleb4]

I tested this on another PC. sorry it took awhile, I had to clone the drive to fix bad sectors. I can confirm that this bug can be replicated on completely different hardware,

[AAndyProgram]

I don't forget this issue. But right now I don't see the error and have no idea how to determine what and where exactly is happening. Unfortunately, both of my home computers don't have HDR. As you said, the problem occurs when the HDR mode is enabled. Right now, I can only suggest you to close SCrawler when you start playing the game.

[appleb4]

Actually it doesn't need HDR, I only mentioned that when I was first trying to figure out how to replicate the bug cause I managed to catch it while my monitor was switching modes. I can replicate it on another non-HDR display, it seems to just need an exclusive fullscreen app, even a video player works for me (MPC-HC) when I switch it to exclusive D3D fullscreen

[AAndyProgram]

I can replicate it on another non-HDR display, it seems to just need an exclusive fullscreen app, even a video player works for me (MPC-HC)

Tried your steps with MPC-HC in full screen mode. No error occurred.

[appleb4]

Tried your steps with MPC-HC in full screen mode. No error occurred.

These are the settings I use, but it seems to happen less often compared to fullscreen games. Of my 4 tests, they all froze only on the second time the autodownloader ran. Also, if D3D fullscreen wasn't enabled before, MPC needs a restart. Fullscreen mode in D3D should make your monitor turn black for a second, and the context menu doesn't work while in fullscreen

[AAndyProgram]

Okay, I'll try again.

[AAndyProgram]

Tried to repeat and nothing. The autodownloader was launched 6 times while playing a video with your settings in full screen mode. 😕

These are the settings I've tried (as far as I can see from the settings interface, we are using different versions of MPC):

There are my current (default) settings:

[appleb4]

Yes my MPC is a little old, 1.9.2 from last year https://github.com/clsid2/mpc-hc/releases

I did try the latest version though, on another PC as well which is a fresh windows 10 install and was able to recreate the issue. I really appreciate the effort you've put in trying to figure this out. Whatever the issue it's definitely something very obscure. Hopefully someone else can come forward with something new, drag083's occurrence had to have been related but clearly we're missing some important info to figure out the cause.

[AAndyProgram]

Yes my MPC is a little old, 1.9.2 from last year https://github.com/clsid2/mpc-hc/releases

No, my player is a little old. 1.7.13 🤣

Hopefully someone else can come forward with something new, drag083's occurrence had to have been related but clearly we're missing some important info to figure out the cause.

I hope the same

[drag083]

Well I've been running the last couple of releases for days on end with exactly the same settings and so far... no locks up.

Will keep an eye on it though and report back if this changes.

[AAndyProgram]

Seriously? I haven't fixed anything on this issue. So in the current (latest release) the problem is gone? Did I understand correctly?

[drag083]

Seriously? I haven't fixed anything on this issue. So in the current (latest release) the problem is gone? Did I understand correctly?

Yes, from what I can tell it doesn't seem to be locking up like it was previously (for me at least). When that was happening, every 2 days (max 3) it would just lock up and become unresponsive. I haven't seen the same behaviour recently (im on 2023.2.5.0) but will keep an eye on it.

[AAndyProgram]

@appleb4, do you still have this problem?

[appleb4]

@appleb4, do you still have this problem?

Just tested latest version, yes it's still happening

[AAndyProgram]

Hello. Any changes?

[drag083]

Just fyi I still haven’t had it lock up in quite some time, seems to be pretty smooth now (am on the latest version).

[AAndyProgram]

@drag083 thank you for your reply 😊

[appleb4]

I can confirm it still happens on the latest version for me. I guess it's easy enough to avoid 🤷🏻‍♂️

Edit: Actually, it happened overnight on my other PC that is a fresh windows install that I just leave running. I'll keep an eye on it, hopefully I can figure more out Editedit: Might have something to do with waking from sleep model, because I saw scrawler being unresponsive 2 times after waking from sleep, maybe I can close scrawler before putting my PC to sleep

Nevermind, not sleep mode, it was unresponsive again when I got home today. Very strange

[AAndyProgram]

Hello @appleb4. Has anything changed? Does this still happening even if you close SCrawler before putting your computer to sleep?

[appleb4]

Hello @appleb4. Has anything changed? Does this still happening even if you close SCrawler before putting your computer to sleep?

Sorry for the late reply. Yes, still happens on two PCs. I'm completely out of ideas, my workaround has just been to run a script to end the task and restart it every hour,