search

Blockstream / satellite

Blockstream Satellite
https://blockstream.github.io/satellite/
GNU General Public License v3.0
977 stars 152 forks source link

—report command line option crash #86

Closed bg002h closed 2 years ago

bg002h commented 2 years ago

My intel nuc has a dual satellite connection to G18 and E113 via an s400 and a base station connection to E113. It’s also on the internet, but only gets bitcoin data via satellite.

When CNR goes too low, and the S400 loses signal lock, the blocksat-cli standalone monitor —report command crashes. Some sort of type casting error.

2022-01-16 11:33:35  Lock = True; Level = -51.00dBm; SNR = 9.10dB; BER = 1.00e+00; Packet Errors = 164; 2022-01-16 11:33:37  Lock = False; Traceback (most recent call last):   File "/usr/local/bin/blocksat-cli", line 8, in     sys.exit(main())   File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/main.py", line 81, in main     args.func(args)   File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/standalone.py", line 616, in monitor     stats = s400.get_stats()   File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/standalone.py", line 284, in get_stats     == '< 70') else float(signal_raw) ValueError: could not convert string to float: ''

bg002h commented 2 years ago

There is a NOTE: in the get_stats function that says the s400 does not return the signal level if unlocked…hence the <70 db check in the log above.

I’ll keep looking.

bg002h commented 2 years ago

Wow, python is weird. The offending code must be here…I added //comments to explain the behavior I’m seeing. 

    signal_lock_raw = res[0][1]
    signal_raw = res[1][1]
    //other stuff

    # Parse
    signal_lock = (signal_lock_raw == 'locked')//this doesn’t do what is intended for some reason
    stats = {'lock': (signal_lock, None)}

    # Metrics that require locking
    #
    # NOTE: the S400 does not return the signal level if unlocked.
    if (signal_lock): //somehow when s400 reports false for signal lock, this code still gets run 
        level = float('nan') if (signal_raw
                                 == '< 70') else float(signal_raw)
        cnr = float('nan') if (c_to_n_raw == '< 3') else float(c_to_n_raw)
bg002h commented 2 years ago

Similar issue in same function (get_stats), but this was while monitoring G18:

2022-01-16 11:33:34 Lock = True; Level = -33.00dBm; SNR = 12.20dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:36 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3;

2022-01-16 11:33:38 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-16 11:33:37 Lock = True; Level = 1.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 1; 2022-01-16 11:33:39 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:41 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:43 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:44 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:50 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:52 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:54 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:56 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:58 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:33:59 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:34:01 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:34:03 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:34:05 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 3; 2022-01-16 11:34:07 Lock = False; Traceback (most recent call last): File "/usr/local/bin/blocksat-cli", line 8, in sys.exit(main()) File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/main.py", line 81, in main args.func(args) File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/standalone.py", line 616, in monitor stats = s400.get_stats() File "/home/bcg/.local/lib/python3.8/site-packages/blocksatcli/standalone.py", line 285, in get_stats cnr = float('nan') if (c_to_n_raw == '< 3') else float(c_to_n_raw) ValueError: could not convert string to float: ''

bg002h commented 2 years ago

Sometimes the code works properly:

22-01-16 12:27:35 INFO Reporting Rx status to https://satellite.blockstream.space/monitoring/metrics

----------------------------- Receiver Monitoring ----------------------------- 2022-01-16 12:27:37 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:39 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:41 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:43 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:45 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:47 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:49 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:50 Lock = True; Level = -33.00dBm; SNR = 12.10dB; BER = 8.94e-08; Packet Errors = 32; 2022-01-16 12:27:52 Lock = False; 2022-01-16 12:27:54 Lock = False; 2022-01-16 12:27:56 Lock = False; 2022-01-16 12:27:57 Lock = False; 2022-01-16 12:27:59 Lock = False; 2022-01-16 12:28:01 Lock = False; 2022-01-16 12:28:03 Lock = False; 2022-01-16 12:28:04 Lock = False; 2022-01-16 12:28:07 Lock = False; 2022-01-16 12:28:08 Lock = False; 2022-01-16 12:28:10 Lock = False; 2022-01-16 12:28:12 Lock = False; 2022-01-16 12:28:14 Lock = False; 2022-01-16 12:28:16 Lock = False; 2022-01-16 12:28:18 Lock = False; 2022-01-16 12:28:20 Lock = False; 2022-01-16 12:28:21 Lock = True; Level = -32.00dBm; SNR = 12.20dB; BER = 8.94e-08; Packet Errors = 35; 2022-01-16 12:28:23 Lock = True; Level = -32.00dBm; SNR = 12.20dB; BER = 8.94e-08; Packet Errors = 35; 2022-01-16 12:28:25 Lock = True; Level = -32.00dBm; SNR = 12.20dB; BER = 8.94e-08; Packet Errors = 35;

bg002h commented 2 years ago

On a performance note, both G18 and E113 are offline right now in my s400…base station pointed at E113 is showing signal 100%, same signal strength of -41.67 or -41.88 db

blockstreamsatellite commented 2 years ago

Hi @bg002h ,

Thanks for the detailed reports. We've just pushed a potential fix at https://github.com/Blockstream/satellite/commit/bf95ef0f464c015d795e31fbfca8cd00737df27d.

To try it, you need to run blocksat-cli directly from the source code, as follows:

  1. Clone the repository and switch to branch s400_fix_lock_parsing:
git clone https://github.com/Blockstream/satellite.git -b s400_fix_lock_parsing
  1. Run blocksat-cli from file blocksat-cli.py at the root of the satellite directory just cloned:
cd satellite/
./blocksat-cli.py standalone monitor --report
bg002h commented 2 years ago

This seems to work! I restarted the monitoring on all three of my satellite connections (2 s400 & 1 base station) and then went to s400 admin console and changed polarity or frequency to cause a loss of lock on the signal and that works fine.

I’ll run outside and stand in front of my 1.2m dish pointed at G18 and the big flat panel pointed at E113 to try to obscure signal and see if I can get low CNR but still stay locked.

The base station is pointed at E113 and shouldn’t be affected by this patch and it’s in the roof (because it’s so light!) and it’s too cold to climb up there anyhow :)

bg002h commented 2 years ago

Brrr…I got CNR under 3 and tripped the error “Report failed: {“snr”: [“Special numeric values (nan or infinity) are not permitted.”]}. And the software didn’t crash!

I think both bugs are fixed.

Strong (and fast!) work on that code!

bg002h commented 2 years ago

Here’s a —log-scrolling view of what happens if I hold my pistol in front of the LNB at varying distances on G18…seems I got both low CNR & loss of lock situations here:

2022-01-17 20:38:54 Lock = True; Level = -33.00dBm; SNR = 12.00dB; BER = 4.76e-04; Packet Errors = 3; 2022-01-17 20:38:56 Lock = True; Level = -35.00dBm; SNR = 9.20dB; BER = 4.76e-04; Packet Errors = 3;

2022-01-17 20:38:59 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:38:59 Lock = True; Level = -39.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 3; 2022-01-17 20:39:01 Lock = False; 2022-01-17 20:39:03 Lock = True; Level = -36.00dBm; SNR = 4.80dB; BER = 1.00e+00; Packet Errors = 6; 2022-01-17 20:39:05 Lock = True; Level = -37.00dBm; SNR = 3.10dB; BER = 1.00e+00; Packet Errors = 6;

2022-01-17 20:39:07 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:39:07 Lock = True; Level = -37.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 6; 2022-01-17 20:39:09 Lock = False; 2022-01-17 20:39:11 Lock = False; 2022-01-17 20:39:12 Lock = False; 2022-01-17 20:39:14 Lock = True; Level = -32.00dBm; SNR = 12.00dB; BER = 1.00e+00; Packet Errors = 9;

2022-01-17 20:39:16 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:39:16 Lock = True; Level = -40.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 16;

2022-01-17 20:39:18 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:39:18 Lock = True; Level = -38.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 19; 2022-01-17 20:39:19 Lock = False; 2022-01-17 20:39:21 Lock = False; 2022-01-17 20:39:23 Lock = False; 2022-01-17 20:39:25 Lock = False; 2022-01-17 20:39:27 Lock = True; Level = -32.00dBm; SNR = 12.10dB; BER = 1.00e+00; Packet Errors = 22; 2022-01-17 20:39:28 Lock = True; Level = -32.00dBm; SNR = 12.20dB; BER = 1.00e+00; Packet Errors = 22; 2022-01-17 20:39:31 Lock = True; Level = -32.00dBm; SNR = 12.00dB; BER = 1.00e+00; Packet Errors = 22;

bg002h commented 2 years ago

For the big flat panel pointed at E113 it took a little more surface area than a pistol and my flashlight so I had to obscure the antenna’s view of the sky with progressively more of my body…apparently I have a shit ton of SNR to spare.

2022-01-17 20:51:19 Lock = True; Level = -50.00dBm; SNR = 11.50dB; BER = 1.51e-03; Packet Errors = 813; 2022-01-17 20:51:20 Lock = True; Level = -51.00dBm; SNR = 10.20dB; BER = 1.51e-03; Packet Errors = 813; 2022-01-17 20:51:22 Lock = True; Level = -51.00dBm; SNR = 8.10dB; BER = 1.51e-03; Packet Errors = 816; 2022-01-17 20:51:24 Lock = True; Level = -52.00dBm; SNR = 7.70dB; BER = 1.51e-03; Packet Errors = 820; 2022-01-17 20:51:25 Lock = True; Level = -51.00dBm; SNR = 7.80dB; BER = 1.51e-03; Packet Errors = 826; 2022-01-17 20:51:27 Lock = True; Level = -51.00dBm; SNR = 7.50dB; BER = 1.51e-03; Packet Errors = 830; 2022-01-17 20:51:29 Lock = True; Level = -52.00dBm; SNR = 7.70dB; BER = 1.51e-03; Packet Errors = 832; 2022-01-17 20:51:31 Lock = True; Level = -52.00dBm; SNR = 7.50dB; BER = 1.51e-03; Packet Errors = 840; 2022-01-17 20:51:33 Lock = True; Level = -53.00dBm; SNR = 7.10dB; BER = 1.51e-03; Packet Errors = 847; 2022-01-17 20:51:35 Lock = True; Level = -53.00dBm; SNR = 4.90dB; BER = 1.51e-03; Packet Errors = 851;

2022-01-17 20:51:37 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:51:37 Lock = True; Level = -54.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 851;

2022-01-17 20:51:38 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:51:38 Lock = True; Level = -54.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 851;

2022-01-17 20:51:40 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:51:40 Lock = True; Level = -54.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 865; 2022-01-17 20:51:42 Lock = True; Level = -53.00dBm; SNR = 3.30dB; BER = 1.00e+00; Packet Errors = 873; 2022-01-17 20:51:43 Lock = True; Level = -53.00dBm; SNR = 4.70dB; BER = 1.00e+00; Packet Errors = 873; 2022-01-17 20:51:46 Lock = True; Level = -53.00dBm; SNR = 4.80dB; BER = 1.00e+00; Packet Errors = 883; 2022-01-17 20:51:47 Lock = True; Level = -53.00dBm; SNR = 4.10dB; BER = 1.00e+00; Packet Errors = 883;

2022-01-17 20:51:49 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:51:49 Lock = True; Level = -54.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 883; 2022-01-17 20:51:51 Lock = False; 2022-01-17 20:51:52 Lock = False; 2022-01-17 20:51:54 Lock = False; 2022-01-17 20:51:56 Lock = False; 2022-01-17 20:51:57 Lock = False; 2022-01-17 20:51:59 Lock = False; 2022-01-17 20:52:01 Lock = False; 2022-01-17 20:52:03 Lock = False; 2022-01-17 20:52:05 Lock = False; 2022-01-17 20:52:06 Lock = False; 2022-01-17 20:52:08 Lock = False; 2022-01-17 20:52:10 Lock = True; Level = -53.00dBm; SNR = 3.00dB; BER = 1.00e+00; Packet Errors = 891; 2022-01-17 20:52:12 Lock = True; Level = -53.00dBm; SNR = 4.00dB; BER = 1.00e+00; Packet Errors = 894; 2022-01-17 20:52:13 Lock = True; Level = -53.00dBm; SNR = 4.70dB; BER = 1.00e+00; Packet Errors = 895; 2022-01-17 20:52:15 Lock = True; Level = -52.00dBm; SNR = 5.90dB; BER = 1.00e+00; Packet Errors = 901; 2022-01-17 20:52:18 Lock = True; Level = -52.00dBm; SNR = 7.40dB; BER = 1.00e+00; Packet Errors = 903; 022-01-17 20:52:24 Lock = True; Level = -52.00dBm; SNR = 8.80dB; BER = 1.00e+00; Packet Errors = 904; 2022-01-17 20:52:26 Lock = True; Level = -51.00dBm; SNR = 10.10dB; BER = 1.00e+00; Packet Errors = 909; 2022-01-17 20:52:28 Lock = True; Level = -51.00dBm; SNR = 10.90dB; BER = 1.00e+00; Packet Errors = 909; 2022-01-17 20:52:30 Lock = True; Level = -52.00dBm; SNR = 8.80dB; BER = 1.00e+00; Packet Errors = 915; 2022-01-17 20:52:32 Lock = True; Level = -53.00dBm; SNR = 4.50dB; BER = 1.00e+00; Packet Errors = 919;

2022-01-17 20:52:33 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:52:33 Lock = True; Level = -53.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 924;

2022-01-17 20:52:35 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:52:35 Lock = True; Level = -53.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 927;

2022-01-17 20:52:37 ERROR Report failed: {"snr": ["Special numeric values (nan or infinity) are not permitted."]}

2022-01-17 20:52:37 Lock = True; Level = -54.00dBm; SNR = nan; BER = 1.00e+00; Packet Errors = 927; 2022-01-17 20:52:39 Lock = True; Level = -50.00dBm; SNR = 7.50dB; BER = 1.00e+00; Packet Errors = 953; 2022-01-17 20:52:41 Lock = True; Level = -50.00dBm; SNR = 11.10dB; BER = 1.00e+00; Packet Errors = 956; 2022-01-17 20:52:43 Lock = True; Level = -51.00dBm; SNR = 11.40dB; BER = 1.00e+00; Packet Errors = 956;

bg002h commented 2 years ago

As best as I can tell, this issue is solved.

bg002h commented 2 years ago

Almost at 48 hours of constant reporting without a crash. I learned how to use “screen” command and how to set scroll back buffer to crazy big…over the last two days I see variation in signal power level on G18 varying over an extreme range of 10 dB ( -49 to -59) without losing signal lock (although there are a few times when lock was lost at -59 dBw). E113 showed much lower range of -31 to -37 dB before losing lock (I have long coax cables and +15 to 20 dB low-pass amplifiers on each line).

The base station doesn’t seem to care about power level. Range has been from -41.67 to -42.29 while signal quality has always been 100%.

blockstreamsatellite commented 2 years ago

Hi @bg002h ,

Thanks for sending the logs. They were very helpful.

The issue you are seeing regarding error {"snr": ["Special numeric values (nan or infinity) are not permitted."]} is really due to the CNR dropping to a low value as you intended in the tests. The S400 receiver does not measure the SNR accurately if it falls below 3 dB. Similarly, it does not measure the signal level if it falls below -70 dBm.

We've reviewed what the CLI does when the S400 returns an indication that either the SNR or signal level is below the limit for being measured. Previously, the CLI would report the value as NaN (not a number). Now, after 4b0ad86, it will no longer report the value. Instead, you will only see a warning on the console. So the error {"snr": ["Special numeric values (nan or infinity) are not permitted."]} should disappear if you pull the latest patch with:

git pull origin s400_fix_lock_parsing

Instead of the error, you will see logs like:

2022-01-24 08:19:45 WARNING  Signal level not accurately measured (below -70 dBm)

or

2022-01-24 08:19:45 WARNING  SNR not accurately measured (below 3 dB)

Aside from that, did you find any other problem? Is the CLI still running and reporting since the last time you checked?

Thanks again

blockstreamsatellite commented 2 years ago

Hi @bg002h ,

We are going to close this issue. The fixes discussed in this thread are now available in the new Blocksat CLI version (0.4.3).

Thanks!