Serious Security Vulnerabilities / Project Abandoned?

MaxwellDPS commented 3 years ago

Hey @dreinhold, not to call you out but, is there any active dev still happening on this project?

There are some serious concerns I have with this app, especially if its gonna be WAN accessible.

Is there any plan to upgrade above Django 1 as there are a handful of CVEs
- CVE-2021-33203 Directory Traversal
- CVE-2021-32052 HTTP Header Injection
- CVE-2020-9402 SQL Injection
- CVE-2019-3498 Content Spoofing
- CVE-2018-14574 Open Redirect
- CVE-2009-3695 Regular Expression Denial of Service (ReDoS)
- THIS IS JUST DJANGO! ^
None of the API's are login restricted
- granted some may want to list talk groups for anon, but if anon access is false, they should be login restricted
AWS S3 Has no signing option to prevent unauthorized access to audio

In the meantime Id seriously recommend no-one put this app on the internet

@dreinhold no shame if you cant maintain it, You created a great project and I love it! However Id seriously recommend you archive and label it with these warnings

Thanks Max :)

bctrainers commented 3 years ago

Any other CVE's to be aware of? Most of the package's / pip's needed for trunk-player can be safely upgraded on the minor version scale without having to fight about with code changes. Things like django core can only be upgraded to version 1.11.29 - granted 1.x series was EOL'd many years ago, with security / lts ending a year and a half ago.

MaxwellDPS commented 3 years ago

@bctrainers Im sure there is a handful on the packages side, if I have time I'll go through and check https://snyk.io/. I was thinking about forking this to a docker centric version and adding federation across systems

MaxwellDPS commented 3 years ago

@bctrainers These are the Vulnerable packages as of 9/11/2021

EDIT: THIS DOESNT INCLUDE SUB-DEPENDENCIES

Vulnerable packages:

channels==1.1.8
- https://snyk.io/vuln/pip:channels@1.1.8 Denial of Service (DoS)
django-allauth==0.34.0
- https://snyk.io/vuln/pip:django-allauth@0.34.0 Account Hijacking
djangorestframework<3.12
- https://snyk.io/vuln/pip:djangorestframework@%3C3.12 XML External Entity (XXE) Injection; x4 Cross-site Scripting (XSS); Information Exposure
django-allauth==0.34.0
- https://snyk.io/vuln/pip:django-allauth@0.34.0 Account Hijacking

dreinhold commented 3 years ago

Yea sadly it's pretty abandoned. As my area went 100% encrypted a few years back and I have no used it in that long, it's been hard to dedicate time on something you just don't use. Always happy to see the few pull requests when they come. I open sourced the project hopping to find others to collaborate with to drive up the developers on this to +1, but never really got one. I even did a could web training for a few people who showed interest in adding some enhancements, to try and drive some new developers in here, but never got it.

I guess enough, it's clear i'm the only one and as i'm not actively doing any development or upgrades I should just bite the bullet and call an official end to this. ;(

MaxwellDPS commented 3 years ago

Well if I have the time Im going to try to fork it and overhaul it a bit, Any thoughts on implmenting site federation from multiple systems?

MaxwellDPS commented 3 years ago

@dreinhold would you be willing to provide a breakdown of how this app works so I can rip and replace old libs with the newer Libs

I am familiar w/ Django and celery but not Daphne and asgi/Websockets; However Im sure I can figure out whats happening w/ enough time

First issue w/ Django 3 is Channels 2 depreciated Group()

dreinhold commented 3 years ago

Yeah I think channels is the only sticking point, i think the rest is going to be minor changes. As you can run it without channels (by just setting the JS to refresh the page which i think many people who never got channels working do anyways), I was thinking maybe the best is do the upgrade without channels as a first step then get the channels working again after.

MaxwellDPS commented 3 years ago

Ok, So from what I've gathered so far on Transmission.Save() send_mesg() is called and then sends a message to the corresponding channel group.

From my understanding changing it to (see below*) should be in spec with channels 2

Then as for asgi.py it needs to be closer to (see below**); where Im getting a little lost is the URL's, does it currently send WS over specific URLs and if so where are those defined?

Following that, the consumers need to be converted to a class. closer to (see below ***). Are you able to ELI5 what these functions (ws_connect, ws_receive, ws_disconnect ) are doing in radio.consumers.py

Also by chance are you able to provide any guidance on converting these consumers per https://channels.readthedocs.io/en/latest/topics/consumers.html#basic-layout

consumers.py ***


class RadioConsumer(WebsocketConsumer):
    def connect(self):
        self.
        self.room_name = self.scope['url_route']['kwargs']['room_name']
        self.room_group_name = 'chat_%s' % self.room_name

        # Join room group
        async_to_sync(self.channel_layer.group_add)(
            self.room_group_name,
            self.channel_name
        )

        self.accept()

    def disconnect(self, close_code):
        # Leave room group
        async_to_sync(self.channel_layer.group_discard)(
            self.room_group_name,
            self.channel_name
        )

    # Receive message from WebSocket
    def receive(self, text_data):
        text_data_json = json.loads(text_data)
        message = text_data_json['message']

        # Send message to room group
        async_to_sync(self.channel_layer.group_send)(
            self.room_group_name,
            {
                'type': 'chat_message',
                'message': message
            }
        )

ASGI.py **

import os
from django.conf.urls import url
from django.core.asgi import get_asgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "trunk_player.settings")
django_asgi_app = get_asgi_application()

from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter

from radio.consumers import RadioConsumer

application = ProtocolTypeRouter({
    "http": django_asgi_app,

    # WebSocket chat handler
    "websocket": AuthMiddlewareStack(
        URLRouter([
            url(r"^chat/admin/$", AdminChatConsumer.as_asgi()),
            url(r"^chat/$", PublicChatConsumer.as_asgi()),
        ])
    ),
})

send_mesg *

@receiver(post_save, sender=Transmission, dispatch_uid="send_mesg")
def send_mesg(sender, instance, **kwargs):
    from channels.layers import get_channel_layer
    channel_layer = get_channel_layer()
    #log.debug('Hit post save()')
    #log.debug('DATA %s', json.dumps(instance.as_dict()))
    #log.error('DATA %s', json.dumps(instance.as_dict()))
    tg = TalkGroup.objects.get(pk=instance.talkgroup_info.pk)
    tg.last_transmission = timezone.now()
    tg.save()
    groups = tg.scanlist_set.all()
    for g in groups:        
        async_to_sync(channel_layer.group_send)('livecall-scan-'+g.slug, {'text': json.dumps(instance.as_dict())})

        #Group('livecall-scan-'+g.slug, ).send({'text': json.dumps(instance.as_dict())})
    async_to_sync(channel_layer.group_send)('livecall-tg-' + tg.slug, {'text': json.dumps(instance.as_dict())})
    #Group('livecall-tg-' + tg.slug, ).send({'text': json.dumps(instance.as_dict())})

    # Send notification to default group all the time
    async_to_sync(channel_layer.group_send)('livecall-scan-default', {'text': json.dumps(instance.as_dict())})
    #Group('livecall-scan-default').send({'text': json.dumps(instance.as_dict())})

dreinhold commented 3 years ago

The WS connections are first passed to the routing.py where if it is a connect it is passed to consumers.py to the ws_connect() method.

In that method it gets the full url path to get the tg/scan group the user is on to register the connection to that group in channels. The when a new transmission is added it looks up all the scan groups that tg is in and fires the messaged into all those groups to let the browser js know there is an updated transmission for a tg they are watching.

MaxwellDPS commented 3 years ago

@dreinhold Thank you! I think im starting to get a hold on whats going on, If you have time can you please review commits below, If im not mistaken should implement channels v2

Edit: btw docker-compose up --build works like a charm on that branch

https://github.com/MaxwellDPS/trunk-player/tree/74ef5acc27a3d28ae3549655552c1a04954a2a00 https://github.com/MaxwellDPS/trunk-player/tree/1170a1bfab4d8b8c2c9641ccda56ac053d4b35ba

MaxwellDPS commented 3 years ago

Likely has some bugs still but Channel2 is implemented on https://github.com/MaxwellDPS/trunk-player/tree/beta

Edit: Scan groups connect

MaxwellDPS commented 3 years ago

So after I do some transmission testing, I'll submit a pull with the upgrade to Channels v2 and that make it so Django and all other deps are more up-to date. This should resolve all of the above vulns above ^.

Then I think it should be easier to stay up on this project. and hopefully save Trunk-Player from the grave

I do plan to fork this with an emphasis on a new UI, Multi-TrunkPlayer Federation, removal of Payment features, I would love to get some advise on all of the above!

dreinhold commented 3 years ago

You are the man, I was sad to say it had died and you just went a brought it back to life. I took a quick look at the changes,all looked good.

Maybe start a new issue for each of the ideas. I'm not sure what the federation you are looking for, like multiple trunk player sites working together, or multiple recording sites feeding trunk player?

MaxwellDPS commented 3 years ago

You're the man, you did the heavy lifting on this one lol. Ill try to get that PR out Tomorrow morning!

And as for Federation I have a mix of use cases for both multiple trunk player sites working together, and multiple recording sites feeding trunk player

MaxwellDPS commented 3 years ago

@dreinhold Shoot, Just realized that the Django Jump from 2-3 Removes legacy python2 support which will break Pinax Stripe, Do you want me to try and fix that or PR w/o Plans and Pinax

dreinhold commented 3 years ago

It looks like Pinax Stripe is also a project on deaths door, so no need to bring it with us. I saw you had removed it from the installed_apps.

I am good dropping support for it (anyone actively using it we will need to find an alternate)

MaxwellDPS commented 3 years ago

Yeah, this is gonna have some breaking changes to the DB... Hmmm, may need to figure out a migration script for legacy users

dreinhold commented 3 years ago

I tagged the current as "django-1.x" so once we pull in your changes if someone is using something that breaks, makes it easier to for them to get to the old.

dreinhold commented 3 years ago

what are you seeing break in the DB.

MaxwellDPS commented 3 years ago

With the removal of Pinax stripe and plans it throws

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 88, in handle
    loader = MigrationLoader(None, ignore_no_migrations=True)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 259, in build_graph
    self.graph.validate_consistency()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 195, in validate_consistency
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 195, in <listcomp>
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 58, in raise_error
    raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration radio.0038_stripeplanmatrix dependencies reference nonexistent parent node ('pinax_stripe', '0007_auto_20170108_1202')

dreinhold commented 3 years ago

Eww, yeah. Thinking about this.

bctrainers commented 3 years ago

Went ahead and synced my fork of TrunkPlayer, it would appear there are many more security vulnerabilities than I thought.

https://github.com/bctrainers/trunk-player/security/dependabot

Keep in mind my fork does have numerous changes that probably break stock-TrunkPlayer that @dreinhold has made.

MaxwellDPS commented 3 years ago

@dreinhold editing the 0038 migration fixed the migrations

MaxwellDPS commented 3 years ago

@dreinhold Just ssubmitted PR #105 likely want to test it a bit but should be GTG. You may want to check the references to Pinax stripe

bctrainers commented 3 years ago

@dreinhold Just submitted PR #105 likely want to test it a bit but should be GTG. You may want to check the references to Pinax stripe

Upon my initial review, things look good - but have not tested it on a machine yet. When I get a moment this afternoon/evening, I will test it on a machine to see if anything explodes ( figuratively speaking :^) ).

Are there any key points to note, or upgrade process notes from the current git version and your pending pull request?

MaxwellDPS commented 3 years ago

So @bctrainers I need some feedback on the migrations on a legacy system. Just spun up an new one and it works flawlessly with a clean DB. As for notes:

if you are using docker double check the new env some boolean 1/0 are now True/False
Also make sure to re do pip install -r requirements.txt
Otherwise it should be just a ./manage.py makemigrations && ./mange.py migrate

MaxwellDPS commented 3 years ago

@bctrainers another quirk you may have to nuke you venv and recreate it

MaxwellDPS commented 3 years ago

@dreinhold are the workers being used, because after the update you have to pass the channels that it will work... so a few problems there lol, but it seems the websockets are connecting w/o them

bctrainers commented 3 years ago

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently.

They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd

bctrainers commented 3 years ago

So @bctrainers I need some feedback on the migrations on a legacy system. Just spun up an new one and it works flawlessly with a clean DB. As for notes:

if you are using docker double check the new env some boolean 1/0 are now True/False

Also make sure to re do pip install -r requirements.txt

Otherwise it should be just a ./manage.py makemigrations && ./mange.py migrate

Looks like a fairly standard process, so that's nice. I don't use docker for TrunkPlayer - It's on a virtual machine. :)

MaxwellDPS commented 3 years ago

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently.

They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd

Hmm, I'm wondering if Channels 1 handled that differently then 2. Not sure how we are going to make that work tbh since the names are relative

dreinhold commented 3 years ago

@dreinhold editing the 0038 migration fixed the migrations

This will cause issues for existing installations. We will need to create a new migration that deletes that column from existing tables also.

dreinhold commented 3 years ago

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently. They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd

Hmm, I'm wondering if Channels 1 handled that differently then 2. Not sure how we are going to make that work tbh since the names are relative

Yes the runworkers changed from 1 to 2 (I don't fully understand what that means for us)

MaxwellDPS commented 3 years ago

@dreinhold editing the 0038 migration fixed the migrations

This will cause issues for existing installations. We will need to create a new migration that deletes that column from existing tables also.

Should this work?


# -*- coding: utf-8 -*-
# Generated by Django 1.11.17 on 2019-01-04 22:55
from __future__ import unicode_literals

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('radio', '0062_siteoption_copyright_notice'),
    ]

    operations = [
        migrations.RemoveField(model_name='StripePlanMatrix', name="stripe_plan")
    ]

MaxwellDPS commented 3 years ago

Yes the runworkers changed from 1 to 2 (I don't fully understand what that means for us)

Yeah per this

Note that runworker will only listen to the channels you pass it on the command line. If you do not include a channel, or forget to run the worker, your events will not be received and acted upon.

I think this means we are out of luck on the workers aside from maybe livecall-scan-default? If you pass * it grabs the names of all the files in the dir, so wild cards dont seem to be a thing. One thought is we have a ./mange.py module that dumps the possible options as empty files named the possible channels in a dir, but this seems clunky

s3m1s0n1c commented 3 years ago

You're the man, you did the heavy lifting on this one lol. Ill try to get that PR out Tomorrow morning!

And as for Federation I have a mix of use cases for both multiple trunk player sites working together, and multiple recording sites feeding trunk player

I would love to feed multiple recording sites into one trunk-player site :)

This would be super useful for my site..

Thanks for bringing this project back to life @MaxwellDPS Unfortunately my Django skills are in the "Enough to cause trouble" category..

I personally would love to see this project kept alive and willing to help where I can..

Thanks Sonic

MaxwellDPS commented 3 years ago

Howdy everyone,

Out of curiosity, looks like the webUI is JS based... Any react devs out there 😉👀

MaxwellDPS commented 3 years ago

@dreinhold so I have been running the new fork in docker for a few days w/o runworker and so far I have not noticed any issues. I'd image a site with heavy load may have some delay... Not sure how to handle yet aside from rewriting for a unicast ish channel and js logic.

May need to completely rework how the WS and JS interact, for the workers

dreinhold commented 3 years ago

Howdy everyone,

Out of curiosity, looks like the webUI is JS based... Any react devs out there 😉👀

+1 on the idea of using react as the frontend.

MaxwellDPS commented 3 years ago

@dreinhold so I have been running the new fork in docker for a few days w/o runworker and so far I have not noticed any issues. I'd image a site with heavy load may have some delay... Not sure how to handle yet aside from rewriting for a unicast ish channel and js logic.

May need to completely rework how the WS and JS interact, for the workers

@dreinhold I have found away to accomplish this via a single uni-com WS channel, instead of diffrent ones we send all new messages out to that channel, then we can use JS for some magic. Then workers are useable too @bctrainers

one problem is this is a security issue for TG restriction and should be resolved but not sure how yet... Expect a PR by EOD

MaxwellDPS commented 3 years ago

one problem is this is a security issue for TG restriction and should be resolved but not sure how yet... Expect a PR by E

Edit: Slightly mitigated the security issue by limiting the WS info pushed to the bare minim. so in theory an and un authenticated / unauthorized user could see the TG name, ScanList Names, I can fix the unauthed user with template tags...

Cant think of a great way we can obscure scan list names and TG names w/o using dynamic channel names... aside from asymmetric crypto and client IDs, but that seems like overkill

dreinhold commented 3 years ago

Right now the channel is just a trigger for the page to re pull the api, so you really don't need to pass/leak any data.

dreinhold commented 3 years ago

As you saw i created a django3 branch for all this until we can show both a new install, and an upgrade will work. Then we can make it master.

We should start new issues at this point for new items/ideas, there is a Discussions thing in github now im not sure how it work but ill enable it, might be a good area where new ideas can be hashed out.

Another big part to this is updating the documentation, we have the markup stuff in docs that gets built into the readthedocs site and the wiki. Need to look those over and edit as needed, for those who don't know django but have used TP would be a big help in updating the documents.

dreinhold commented 3 years ago

A big thanks to @MaxwellDPS for just starting the conversation and for the updated code. I have asked him if he would willing to be a committer in the project and he has said yes, so I won't be a bottle neck for any changes going in.

I will be around, and still plan on reviewing the changes I can and jumping in where I can.

Dylan

bctrainers commented 3 years ago

While we're on the basis of a new version of TrunkPlayer, what are our current thoughts with supporting MySQL/MariaDB in addition to PostgreSQL?

FWIW, pg13 works without much issues on TrunkPlayers current code. So that's always good.

My apologies for not replying sooner on testing, real life has been getting in the way. :)

RadioTechGuy commented 3 years ago

Just wanted to say this is so great to see. I hope to also get some time to play!!!!

MaxwellDPS commented 3 years ago

While we're on the basis of a new version of TrunkPlayer, what are our current thoughts with supporting MySQL/MariaDB in addition to PostgreSQL?

FWIW, pg13 works without much issues on TrunkPlayers current code. So that's always good.

My apologies for not replying sooner on testing, real life has been getting in the way. :)

@bctrainers @dreinhold correct me if Im wrong, but shouldn't setting the Django backend to mySQL work out of the box... may also need to install the pip package mysqlclient

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'trunk_player', # Database Name
        'USER': 'trunk_player_user', # Database User Name
        'PASSWORD': 'fake_password', # Database User Password
        'HOST': 'localhost',
        'PORT': '3306',
    }
}

MaxwellDPS commented 3 years ago

Hey all,

Just tested an upgrade of a clean install and @dreinhold the 0063_stablize.py migration crashes with a key error... I think we are ok to orphan stripe_plan, we can always write a postgresql command to clip it. If we remove that migration the upgrade is smooth, but need some brave volunteers with backups ;). TBH I broke my old site (Older Revision) but just connected the DB to the new code and its been running smooth

Upgrade Steps:

Stop trunk player
cd trunk-player
git checkout django3
pip install -r requirements.txt
./mange.py migrate
Restart Trunk player

kcwebby commented 3 years ago

FYI -- I run about 220 Recorders into one Trunk-Player instance. Scaling is difficult, I had to separate out my database platform (postgres) onto separate bare metal, and still run that hardware at about 75% CPU all of the time with 15-30 active database sessions going at once.

The database is certainly the slowest part of the interface as you scale up. Especially when you go into admin and try to create/add the scan lists (something to do with the way it loads all of the talkgroups into the selection dialog) will timeout every time for us (about 80K talkgroups) We actually had to build a small instance and remove all the timeouts to be able to do those kind of data operations (create/edit new scanlists); I'm not sure how the query is assembled to build that table, but I'm assuming there has to be some kind of a loop, with a single query that is executing for every talkgroup. I think that query could be improved, if that's even possible with Django.

I'm one of the people who attended Dylan's training sessions. I was working on developing ideas and trying to learn how django works, but the hill was too steep for me. I was working with Rachel O. Who suddenly and unexpectedly passed away, and haven't had anyone else to work with since. I don't really want the project to die, but as others have said there hasn't been any development in a long time, and I don't currently have enough knowledge to make the wide scale changes that I feel need to be made to enhance the operation and bring new features.

I'm happy to participate in helping others.

On Wed, Sep 15, 2021 at 9:03 PM Max Watermolen @.***> wrote:

Hey all,

Just tested an upgrade of a clean install and @dreinhold https://github.com/dreinhold the 0063_stablize.py migration crashes with a key error... I think we are ok to orphan stripe_plan, we can always write a postgresql command to clip it. If we remove that migration the upgrade is smooth, but need some brave volunteers with backups ;). TBH I broke my old site (Older Revision) but just connected the DB to the new code and its been running smooth

Upgrade Steps:

Stop trunk player

cd trunk-player

git checkout django3

pip install -r requirements.txt

./mange.py migrate

Restart Trunk player

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ScanOC/trunk-player/issues/104#issuecomment-920516546, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSNSAS5J3GRSLBSSX2ML3LUCFF53ANCNFSM5D3DIIAA .

MaxwellDPS commented 3 years ago

@kcwebby Wow, That sounds like a hell of system! 😄 So I can change the talk group fields to raw ID field, which means you'd need to know its ID in the DB but it will load a hell of a lot better, I think django has a UI even with that to help pick and loads in a separate window when needed. I can implement that as a setting in settings.py or via the env... TBH I want to try to make this as Mirco-service/cloud native as possible, so I will be putting a lot of focus towards docker deployments. I would be more than happy to help with getting docker instances stood up too as it seems to be way easier to scale and very reliable.

Also Its a hell of a learning curve but with that big of a system @kcwebby Kubernetes could really make your life easier in the long term.

ScanOC / trunk-player