hcarter333 / rm-rbn-history

Mapping reverse beacon spots and QSOs of the Rockmite 20 at KD0FNR
0 stars 0 forks source link

scraper fails on multiple bands #85

Closed hcarter333 closed 8 months ago

hcarter333 commented 10 months ago

The scraper fails when only QSOs outside of 20 meters were made.

image

message from Actions: image

is a little bit misleading because it doesn't reveal that the curl command was for KO6BTY

hcarter333 commented 10 months ago

The following }{ pair is throwing off the json reader in kepler.py on line 80

 "1727480844": [
    "KP3CW",
    "28057.0",
    "KD0FNR/6",
    7,
    15,
    "2209z 14 Jan"
  ]
}
{
  "1728247515": [
    "N9CO",
    "14057.5",
    "K3Y/6",
    5,
    16,
    "1305z 16 Jan"
  ],
hcarter333 commented 10 months ago

Put this code back in later

        curl 'https://www.reversebeacon.net/spots.php?h=fb94a8&cdx=K3Y/6&s=0&r=100' | jq '.spots[]|= .[0:6]|.spots ' >> incidents.json
        curl 'https://www.reversebeacon.net/spots.php?h=fb94a8&cdx=KO6B*&s=0&r=100' | jq '.spots[]|= .[0:6]|.spots ' >> incidents.json
        curl 'https://www.reversebeacon.net/spots.php?h=fb94a8&cdx=KD0F*&s=0&r=100' | jq '.call_info ' > stations_geo.json
        curl 'https://www.reversebeacon.net/spots.php?h=fb94a8&cdx=K3Y/6&s=0&r=100' | jq '.call_info ' >> stations_geo.json
        curl 'https://www.reversebeacon.net/spots.php?h=fb94a8&cdx=KO6B*&s=0&r=100' | jq '.call_info ' >> stations_geo.json
hcarter333 commented 10 months ago

Use this to remove the curly bracves;

sed '/\}/d'

Don't use a backslash for } for whatever reason

sed '/{/d'

hcarter333 commented 8 months ago

The above comments resolved the issue. Essentially, we were getting more than one data structure when the underlying analysis expected only one. Removing the curly braces flattened the data. At that point everything worked again.