aircraft.json as an alternative input source

[TomMuc1]

there are some points that led me to the conclusion the aircraft.json file from dump1090 would be the much better source

• every second updated and written into raspi memory
• already combined data as all messages hold only parts of information
• mlat data included
• flightaware anonymized mlat data included
• much less cpu-power needed for running daemon-spotter.php (50-80% of one pi core now)

this way you have in one single file ALL data of ALL aircrafts EVERY SECOND

in addition you could split up the tasks of spotter-daemon.php into at least two daemons. one that forwards data to the web-interface and one that writes to database where needed. that would better scale computing power while using different cores for the different tasks.

i'd really encourage you to have look into: https://github.com/mutability/dump1090/blob/master/README-json.md

This file contains dump1090's list of recently seen aircraft. The keys are:

• now: the time this file was generated, in seconds since Jan 1 1970 00:00:00 GMT (the Unix epoch).
• messages: the total number of Mode S messages processed since dump1090 started.
• aircraft: an array of JSON objects, one per known aircraft. Each aircraft has the following keys. Keys will be omitted if data is not available.
• hex: the 24-bit ICAO identifier of the aircraft, as 6 hex digits. The identifier may start with '~', this means that the address is a non-ICAO address (e.g. from TIS-B).
• squawk: the 4-digit squawk (octal representation)
• flight: the flight name / callsign
• lat, lon: the aircraft position in decimal degrees
• nucp: the NUCp (navigational uncertainty category) reported for the position
• seen_pos: how long ago (in seconds before "now") the position was last updated
• altitude: the aircraft altitude in feet, or "ground" if it is reporting it is on the ground
• vert_rate: vertical rate in feet/minute
• track: true track over ground in degrees (0-359)
• speed: reported speed in kt. This is usually speed over ground, but might be IAS - you can't tell the difference here, sorry!
• messages: total number of Mode S messages received from this aircraft
• seen: how long ago (in seconds before "now") a message was last received from this aircraft
• rssi: recent average RSSI (signal power), in dbFS; this will always be negative.

[Ysurac]

Lot of data is write on sd card if an external http server is used mlat data can be send via 30003 if properly configured This will use the same CPU as TCP, maybe more because of json parse.

No real need to use separate script. This is not really CPU intensive if FlightAirMap/MySQL is properly configured. I don't understand how you can have a high CPU usage, even the demo use about 10% CPU of a little VM.

Maybe I will support aircraft.json but really this shouldn't be used. It's not real time and will be CPU intensive and IO intensive.

[TomMuc1]

did you ever ran top on your pi while about 250-300 aircrafts are in sight of your site? the mysql task is about 5-10% cpu and the spotter-daemon.php about 80%.

i wrote a little php script that makes use of the aircraft.json. if i run it to insert all data - i get about 15,000,000 table rows of distinct position data per day with average of 175 aircrafts over the day. the script then uses way below 80% cpu. the json parse takes about 5 milliseconds on a raspberry.

p.s. but don't get me wrong - of course i run the mysql on a different machine and do not write 15 million table rows a day to the pi :)

[TomMuc1]

are these two your feeds? http://flightaware.com/adsb/stats/user/YcaFlight#stats-14817 if so it's no wonder that your raspi-cpu-load is lower: http://flightaware.com/adsb/stats/user/TomMuc

[Ysurac]

It's one of my feeds, but you can see on http://real.flightairmap.com/ that I have more feeds than that.

I'm running it now with 700 flights on the Raspberry PI 3. MySQL use about 5-20% CPU. PHP is always about 5-7%. MySQL and PHP is on the Raspberry PI.

So really I have no idea why it can use so much CPU. There nothing that can be CPU intensive in fact. But with 700 aircraft this use more memory, about 20% of the 1Go available on the Raspberry PI 3. If less memory is available to the system (you can split memory with the GPU), then this can use swap and this will be slow, CPU intensive,...

I always do my test with this feeds, I don't think many have more than that with an antenna.

[TomMuc1]

i have to look into this memory swapping thing! thanks for the suggestion :) moreover i think it's not just a 'how-many-aicrafts' thing but more a 'how-many-messages' thing. my site has a quite good range - so as a result i receive about 100 million raw-messages a day with a single antenna. you see that even the small chunk flightaware uses results in 5-6 times more messages than your sites have - while my aircrafts per day are just about two times of yours.

again - wish you a nice evening we have dinner now cheers tom

[Ysurac]

True, no idea how many messages I receive with the 700 flights feeds, 24kBit/s for now. I don't use my antenna for testing, I don't have enough flights, I use others feeds. spotter-daemon.php check all coordinates if speed, distance,... is realistic. Many messages send false results. This also calculate speed and heading if not available.

When data are added in DB, airline, aircraft name, owner, arrival and beginning airports and schedules are added. It's why DB can be used quite a lot. MySQL can be configured to use less CPU (and more memory) or postgreSQL can be used (seems to use less CPU and less memory).

So php script do a lot more than only parsing data. FlightAirMap is not a map for Dump1090, it can do a lot more than that. It can use Dump1090 as source and many others sources.

You can also try raw format, but I think it's even more CPU intensive (port 30002 on dump1090). I will add aircraft.json support on monday, but I think this will use more CPU than TCP.

[TomMuc1]

i'll answer this tomorrow :) but for now i have to say i'm really impressed how smooth, snappy and good locking your app works even on a 3 year old ipad mini retina. this is awesome! and miles ahead the new flightaware skyview map or virtualradarserver looks and is usable.

i'm really happy that i did not gave up and with your help now have running flightairmap!!!

[TomMuc1]

https://discussions.flightaware.com/t/one-hell-of-a-ride-flightairmaps-the-way-better-virtualradarserver/26122

[TomMuc1]

regarding message-rate and cpu-load: over the day i see between 70-100% cpu for daemon-spotter.php while having about 1400-1500 messages per second according to my dump1090 web-interface. tonight when message rate was down to 500-600 m/sec the cpu-load from daemon-spotter.php went down to about 30%.

could you do a little test for me on your site please. is one of the flightaware feeds from you at your house? if so could you have a look into the dump1090 web-interface and tell me how many msg/sec you see there. and if message rate is anywhere between 500-1000 msg/sec then connect your raspi running flightairmap to only this one source and see what 'top' gives cpu-load for daemon-spotter.php ...

[TomMuc1]

on the other hand i could comment out the console output from daemon-spotter.php and set some microsecond-timers between the different stages of your scrip and see where the biggest load is ...

but first it would be most interesting what cpu-load you get with a reasonable message rate

[TomMuc1]

p.s. at the moment my pi2 running flightairmaps writes about 4mb per hour to flightairmap database

[TomMuc1]

regarding the aircraft.json data-source. i had the same concerns that you have - but when looking into obj documentation of his dump1090-mutability i found so much advantages. for the data written to database it has zero relevance if the insert happens a second sooner or later. for the webinterface that shows the aircrafts on the map - this tiny delay that is good enough for dump1090 own output should be good enough for flightairmap too :) the built-in dump1090 webinterface runs exactly from this aircraft.json file that dump writes every second only into ram of the pi where dump runs. don't confuse this with the aicraft.json history files that dump1090 writes to filesystem/sd-card!

moreover the aircraft.json holds already only (at least to a certain level) valid message data. so you get in one single file all data from all aircrafts in sight of your site within the last second. that means no more combing of messages needed, no more message validation needed, no more duplicates to sort out etc. etc. i'm sure this will lower cpu-load of daemon-spotter.php a lot.

i was wondering if you are aware that dump1090 is not only a very cool mode-s message decoder - but in addition a perfect collector/combiner. you can run dump1090 in net-only mode without a dongle and then input many datasources/datastreams in parallel (e.g. via socat) and get an output where duplicates and non-valid messages are already sorted out. this output can be beast, json or basestation format.

i'm sure the more you dive into the aircraft.json file-thing - the more you will love it :)

p.s. the beast binary format is great - but decoding in php - i'm not sure whether this makes really sense ...

[Ysurac]

Nothing is really done in daemon-spotter.php, all is done in require/class.SpotterImport.php I will see if I can optimize/disable/make as option some part next week.

There is some option to limit data writes in DB, like $globalCoordMinChange and $globalSourcesupdate if you run it as dameon. If you run is a cronjob don't run it too often (and you shouldn't run it as cron, this features is only when there is no other choices).

I'm no more available for today, sorry. Good week end.

[TomMuc1]

have a good one :)

[Ysurac]

I already combine feed with dump1090, I know it quite well. FlightAirMap can combine many more feeds format. I will add aircraft.json, but validation is not enough for FlightAirMap. As I already said, FlightAirMap is not another map for dump1090, else a single page with js would be enough.

[TomMuc1]

now that this great app runs happily 24/7 on my old raspi2 i saw some tiny bugs. now my question is are you interested in that i post them? and if so - i was thinking maybe it would be better if i open a more general issue e.g. 'list of smaller bugs' (or another title you want) - instead of openening an issue for every single tiny issue - and this way other users could post their tiny issues there too ...

[Ysurac]

You can open an issue yes.