DIRWATCH is "hanging" and requires restart of RDIO

[kcwebby]

Dirwatch seems to be hanging while processing the files in the dirwatch folder. Stopping and restarting RDIO and it will pick the files up and process them for an interval of time (sometimes 5 minutes, sometimes hours) Disabling, and Reenabling the dirwatch will also have it pick up the files.

I cannot isolate the issue it is happening on multiple dirwatch only machines.

Is there a way I can obtain some debug logs to see what is happening?

[chuot]

Those dirwatch are monitoring files generated by which software?

Next time it happens, please kill -3 the process to generate a stack trace and attach it to this issue.

[kcwebby]

Thanks -- dirwatch generated by proscan (windows) this is rdio for windows (6.5.3) Is there a way to get a kill -3 in windows?

[chuot]

It's harder on Window, it requires the EXE to be compiled with the pprof package for debuging. I'm already in the process of troubleshooting by having my own ProScan setup, but it takes time.

[kcwebby]

If you want to provide a exe for me to swap out, I'm happy to help troubleshoot. Like I said, you can still stop the service fine, and the interface still responses (slowly) but the dirwatch is suspended until a disable/enable in the admin interface, or you restart the service physically.

[UberPlexCa]

I'm having the same thing on the Linux version. If it happens again I will kill -3

[chuot]

v6.5.4 has a couple of changes on dirwatch, please try and see if it still hangs.

[hoffadg]

Seeing the same thing on v6.6.3 Windows.

[chuot]

@hoffadg I need logs to troubleshoot this one. Is it happening around the same time during the nights? Is there any software scanning the dirwatch folder for viruses? I saw some reports about that on the fsnotify repo, fsnotify is the package I'm using for the dirwatch feature.

[batsmen-mental]

I have also run into this issue multiple times. I checked the database logs and I see I have the issue after the following message is recorded. is in place of the actual json file. dirwatch.ingest: unexpected end of JSON input,

Next time this happens, do I need to run a kill -3?

[a-pir1]

I am also having the same issue. The dirwatch reports that it can't find the file specified and then totally hangs. RDIO is still running though, as my logs then proceed to get filled with "database purging" entries. It just proceeds to ignore all audio ingests unless I restart the application entirely (which I have to now do daily using task scheduler).

[a-pir1]

My current working theory now is: since I need to have a delay set in RDIO before the ProScan recordings are imported into RDIO to prevent them from being uploaded prematurely uploaded, those timers are causing a memory leak potentially which may be what is causing RDIO to hang and need a full restart.

To test this, I tried keeping my exact same setup with the exact same settings, except setting the import delay to zero. When I did this for 7 days, my recordings imported with practically no viable audio, but RDIO was incredibly stable. I also tested this further by setting the import delay to one hour. RDIO crashed far quicker than I usually see.

As far as I can tell, there is no viable workaround for eliminating the import delay while ensuring only full recordings are uploaded. Cooperation from ProScan would be needed. They would likely need to either provide users an option to create the recordings in a temp directory and move them into the recording directory to be immediately imported by RDIO, or they would need to establish a method to directly upload them to RDIO.

I will continue to investigate this issue and a workaround on my end. Please let me know if any of my findings have any viability. :)

[UberPlexCa]

My current working theory now is: since I need to have a delay set in RDIO before the ProScan recordings are imported into RDIO to prevent them from being uploaded prematurely uploaded, those timers are causing a memory leak potentially which may be what is causing RDIO to hang and need a full restart.

To test this, I tried keeping my exact same setup with the exact same settings, except setting the import delay to zero. When I did this for 7 days, my recordings imported with practically no viable audio, but RDIO was incredibly stable. I also tested this further by setting the import delay to one hour. RDIO crashed far quicker than I usually see.

As far as I can tell, there is no viable workaround for eliminating the import delay while ensuring only full recordings are uploaded. Cooperation from ProScan would be needed. They would likely need to either provide users an option to create the recordings in a temp directory and move them into the recording directory to be immediately imported by RDIO, or they would need to establish a method to directly upload them to RDIO.

I will continue to investigate this issue and a workaround on my end. Please let me know if any of my findings have any viability. :)

So I too, used to use Proscan with a program called directory monitor, once proscan released the file lock, Directory monitor would trigger a batch file to upload the call to Rdio watch directory.

BUT

http://www.scannerbox.us/TrunkingRecorder/

The dev of this, which runs in windows has put the Rdio API system into it. This app can watch the Proscan recording directory, after proscan releases the file lock it can UPLOAD using the API. So not only do you get a piece of software that makes an awesome playback archive system for Proscan, you can also upload to Rdio. I personally have replaced my entire analog upload to Rdio with it.

The WebUI for my system is running here https://record.renfrewcountyscanner.com/ You don't need to expose it to the internet but I choose to do it. In the Receiver dropdown if you pick BCT15X, that is the scanner I use to upload to Rdio.

Auto population etc.. all works, goes by the names you setup in Proscan.

I thought I would share this as it makes a very nice addon for Rdio users to pipe in scanner comms.

[a-pir1]

My current working theory now is: since I need to have a delay set in RDIO before the ProScan recordings are imported into RDIO to prevent them from being uploaded prematurely uploaded, those timers are causing a memory leak potentially which may be what is causing RDIO to hang and need a full restart.

To test this, I tried keeping my exact same setup with the exact same settings, except setting the import delay to zero. When I did this for 7 days, my recordings imported with practically no viable audio, but RDIO was incredibly stable. I also tested this further by setting the import delay to one hour. RDIO crashed far quicker than I usually see.

As far as I can tell, there is no viable workaround for eliminating the import delay while ensuring only full recordings are uploaded. Cooperation from ProScan would be needed. They would likely need to either provide users an option to create the recordings in a temp directory and move them into the recording directory to be immediately imported by RDIO, or they would need to establish a method to directly upload them to RDIO.

I will continue to investigate this issue and a workaround on my end. Please let me know if any of my findings have any viability. :)

So I too, used to use Proscan with a program called directory monitor, once proscan released the file lock, Directory monitor would trigger a batch file to upload the call to Rdio watch directory.

BUT

http://www.scannerbox.us/TrunkingRecorder/

The dev of this, which runs in windows has put the Rdio API system into it. This app can watch the Proscan recording directory, after proscan releases the file lock it can UPLOAD using the API.

So not only do you get a piece of software that makes an awesome playback archive system for Proscan, you can also upload to Rdio. I personally have replaced my entire analog upload to Rdio with it.

The WebUI for my system is running here https://record.renfrewcountyscanner.com/ You don't need to expose it to the internet but I choose to do it. In the Receiver dropdown if you pick BCT15X, that is the scanner I use to upload to Rdio.

Auto population etc.. all works, goes by the names you setup in Proscan.

I thought I would share this as it makes a very nice addon for Rdio users to pipe in scanner comms.

So I have been using this in prod now for 3 months and it has been nothing but fantastic. Works like a dream!