Uploading to OpenMHz

[robotastic]

It is now possible to directly upload files to http://openmhz.com. Details coming soon. @porcej has been doing a beta run and it seems to be working. Here are some questions and answers:

• I am now getting an 'SSL Exception: short read' on upload, however, the upload seems to be working just fine. That is sort of a false error. I think I am terminating SSL connections incorrectly. I will try to fix that, but everything seems to work even with this error.

• Do you plan to support radio aliases? Do you mean coming up with aliases for the different radio ids? If so, I am. I had it work previously and just have to port it to work with a multiple systems.

• On openMhz, is there any plan to make complex filtering? (select multiple talk groups, or say 'Police' and 'Fire' groups? That should be there now. If you click on Filter in the upper left hand corner, it should display a pop-up which lets you either select multiple talkgroups or larger groups. Let me know if that is not working or if you were thinking of something else.

• On open MHz, is there a plan to allow someone to listen to multiple radio systems at once (IE, DC Fire, Alex, Fairfax, and Arlington) without having multiple sessions? Not right now. I think the backend API could support it. I would need to come up with a new front end though. I might be game for giving a try if lots of people start uploading to the site.

• Is there a repo for the web player? Not right now. I am still trying to figure out if I could build this into something cool

This is all great stuff! Thanks for all your work!

Thanks!

[imagesafari]

Very cool. Tempted to try this out. We've got 40+ dongles feeding into a version of Dylan's ScanOC .. a lot of traffic :)

[porcej]

• Do you mean coming up with aliases for the different radio ids? If so, I am. I had it work previously and just have to port it to work with a multiple systems.

Yes.

• On Filtering, I should have been more observant. Talkgroups do allow for multiple selections. If groups would allow multiple selections as well that would be useful.

[Treehouseman]

I don't know about a centralized site. That's a lot of data to manage along with the need to add a ton of sort groups and whatnot. Your SmartNet Player repo you have active is at least some form of the openmhz website, I wasn't fully functional but I made it work with enough beating and have it running locally supporting the Mecklenburg and Cabarrus county systems here in NC, call 1,000,000 was actually uploaded to it yesterday. I'm honestly still hesitant about making it public outside of the local circle of people who have been involved in setting it up because it would take a simple config change for the p25 systems to go fully encrypted. After the Keith Scott shooting in Charlotte this past September I had a ton of people requesting the audio, one person even went as far as to attempt to get the audio off the scanner themselves physically. That's not something that needs to get released to the public when protests are still going on.

On another note, if you want to see my code and the changes I made to support multiple systems (4 systems 2 of which have 2 sites), as well as a few other tweaks I'm sure I made I can go and isolate them. Unfortunately though the system separation is a bit manual, basically talk groups have a prefix of the nac and requires that the scanner appends it as well first.

[Treehouseman]

It's possible I might have broken something, uploads were working fine but suddenly the call list isn't loading and I'm getting ssl 504 errors when it tries to upload something.

I was pretty certain I was uploading stuff in the same exact fashion the normal code does, I eliminated any new json data I added and returned the original filenames. other than that everything else is untouched from what you setup.

[patrickfish]

I've been getting 504 errors as well...last successful upload looks like it was ~ [2017-01-23 20:02:59.737041] (US/Pacific).

On Mon, Jan 23, 2017 at 8:17 PM, Treehouseman notifications@github.com wrote:

It's possible I might have broken something, uploads were working fine but suddenly the call list isn't loading and I'm getting ssl 504 errors when it tries to upload something.

I was pretty certain I was uploading stuff in the same exact fashion the normal code does, I eliminated any new json data I added and returned the original filenames. other than that everything else is untouched from what you setup.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274705707, or mute the thread https://github.com/notifications/unsubscribe-auth/ADUMhadCkAqrqH-0o8Je3ULzssRHfh-Sks5rVXs8gaJpZM4Lk9kF .

[Treehouseman]

Looks intermittent, looking at my logs, at least in the beginning, my last successful upload was at 11:03, but when it first started failing it was a 500 error and not 504.

Either it was overloaded or it got some bad data

[Treehouseman]

Calls are loading on the website now, looks like a good bit of the calls if not all were actually uploaded reguardless of the 504 errors.

[robotastic]

Right now I am not doing a good job of tracking 504 and other http responses. I will add in some logging and see if I can find a pattern. The CPU usage is still extremely low.

Is this happening right around the top of the hour? That is when I generate the statistics, but in testing, it doesn’t look like it takes much cpu and finishes in 10 seconds or so.

On Jan 24, 2017, at 12:11 AM, Treehouseman notifications@github.com wrote:

Calls are loading on the website now, looks like a good bit of the calls if not all were actually uploaded reguardless of the 504 errors.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274711617, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53N_cI2tRMetnkNHSYi9i469LhP3qks5rVYgagaJpZM4Lk9kF.

[bbene]

I just set up a system to connect to it and have been getting the 500 errors. It does say HTTPS Upload result: 1, but I don't see any when I view it on OpenMHZ.

[Treehouseman]

it has to do more with the upload structure, despite the errors it's seemingly uploading the calls, the viewing function in the web browser hasn't seemed to have any issues other than last night where it refused to load the call list.

[robotastic]

If you just started uploading, it would probably show up on this page: https://beta.openmhz.com/systems https://beta.openmhz.com/systems The system only checks for active system once an hour.

I will try to get rid of those false HTTP statuses, and provide something more useful.

On Jan 24, 2017, at 8:40 PM, Ben Benesh notifications@github.com wrote:

I just set up a system to connect to it and have been getting the 500 errors. It does say HTTPS Upload result: 1, but I don't see any when I view it on OpenMHZ.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274995187, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53KiX8JAz_Hl6wW4prPKh_AYCYgUAks5rVqgIgaJpZM4Lk9kF.

[bbene]

Alright, I see the system there (beta) but with no recordings loading in the list. I'll give it a bit more time to see if they show up, otherwise something is off.

[Treehouseman]

they should be instantly showing up provided you have it configured properly.

[robotastic]

Something could totally be off :)

Are there .m4a files showing up in the directory on the server you are recording things on? FFMPEG needs to be installed.

On Jan 24, 2017, at 8:46 PM, Ben Benesh notifications@github.com wrote:

Alright, I see the system there (beta) but with no recordings loading in the list. I'll give it a bit more time to see if they show up, otherwise something is off.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274996091, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53AcRpVCnG-hVSWVU7V10MiI7lssJks5rVqlsgaJpZM4Lk9kF.

[bbene]

Ah, I figured it out. In the site I had it with a capital letter for the short name and config had it lowercase. I matched them up and see calls now.

[robotastic]

ah! That is something I should probably handle on my side. I will go add making sure everything is case agnostic to the list of things to work on. Good catch, thanks!

On Jan 24, 2017, at 8:51 PM, Ben Benesh notifications@github.com wrote:

Ah, I figured it out. In the site I had it with a capital letter for the short name and config had it lowercase. I matched them up and see calls now.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274996835, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53CzH1UX3VzP8krdO3dB3Q82MeklUks5rVqqFgaJpZM4Lk9kF.

[bbene]

No problem. I'm a developer by trade so something like that caught my eye and I gave it a try. :)

[Treehouseman]

@robotastic Did you ever see my tip on gitter about having mongodb create an index for the transcode collection on the time variable to speed up loadtimes? When my version of smartnetplayer got around 1M calls it started taking 3-4 seconds to load a page of calls, after making the index, it took <1/4s

I'm not sure if you're still storing everything in one central table (At this point though having multi system I doubt it) but when you start feeding mongo massive amounts of records, it really does slow down.

[robotastic]

@treehouseman - Whoa! I totally missed all that traffic on Gitter. doh! I will go read through the backlog. Actually, right now it is still one big table, but it is index. It might make sense to split it into a separate collection for each system.

On Jan 24, 2017, at 8:54 PM, Treehouseman notifications@github.com wrote:

@robotastic https://github.com/robotastic Did you ever see my tip on gitter about having mongodb create an index for the transcode collection on the time variable to speed up loadtimes? When my version of smartnetplayer got around 1M calls it started taking 3-4 seconds to load a page of calls, after making the index, it took <1/4s

I'm not sure if you're still storing everything in one central table (At this point though having multi system I doubt it) but when you start feeding mongo massive amounts of records, it really does slow down.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-274997339, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53NKcMYRC58Z71yZCp5ULzOf8qRwUks5rVqtNgaJpZM4Lk9kF.

[Treehouseman]

Yeah, we got to be pretty active on gitter, tried to keep loose conversation and quick conversation on it to help eliminate email spam like github issues do.

[Treehouseman]

My desktop at home seems to be the only thing that'll let me access https://admin.openmhz.com, if I try it on my laptop or my phone it just instantly redirects me to https://beta.openmhz.com. Does this have to do with cookies (chrome account linking)?

[robotastic]

I updated the nginx config. I think it should be fixed now. Let me know if it is not.

[patrickfish]

Not sure if it's related, but it looks like a bunch of audio from yesterday went missing.

1/25/2017 before ~1500 US/Pacific results in a "Cannot GET /audio/Davis/2017/1/25/32592-1485384695.m4a"

On Thu, Jan 26, 2017 at 6:36 PM, Luke Berndt notifications@github.com wrote:

I updated the nginx config. I think it should be fixed now. Let me know if it is not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-275575735, or mute the thread https://github.com/notifications/unsubscribe-auth/ADUMhUtmqS2DxcOwWlZeBTjuvjQNTMv8ks5rWVg4gaJpZM4Lk9kF .

[robotastic]

Doh - I thought I fixed that. I tried to do a big switch out to make things case insensitive for the shortnames on systems. I will go check.

On Jan 26, 2017, at 10:20 PM, patrickfish notifications@github.com wrote:

Not sure if it's related, but it looks like a bunch of audio from yesterday went missing.

1/25/2017 before ~1500 US/Pacific results in a "Cannot GET /audio/Davis/2017/1/25/32592-1485384695.m4a"

On Thu, Jan 26, 2017 at 6:36 PM, Luke Berndt notifications@github.com wrote:

I updated the nginx config. I think it should be fixed now. Let me know if it is not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-275575735, or mute the thread https://github.com/notifications/unsubscribe-auth/ADUMhUtmqS2DxcOwWlZeBTjuvjQNTMv8ks5rWVg4gaJpZM4Lk9kF .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/85#issuecomment-275580605, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG53KtwiT6ZAuXNPcrz9BuU2G23PeVjks5rWWJ8gaJpZM4Lk9kF.

[UnderwaterHockey]

This OpenMHZ site is pretty awesome, thanks for all your hard work and hosting!

I added two systems, both on Minnesota's statewide p25 system. Partially for listen-ability, and partially to reduce unnecessary traffic to and from the server, I have been playing with filtering unwanted data, but even when I set the priority of unwanted talkgroups to 100, it's still recording and uploading them. I made sure both the local and remote lists matched (with hex stripped out on remote) and even when playing from the site it shows their priority as 100.

Also, yesterday I was able to filter by "Group" but today I can't seem to get that to function. Is it working for others?

[UnderwaterHockey]

It appears it's my issue, I fixed one of the two by re-doing both local and server side .csv files. Tomorrow I'll try to fix the other.