[feature sugestion] ADS-B / External data parser

[jmceara]

Hi! I was changing somethings in my FAM and I have an idea (and built!). It's an specific external parser that could work without other libs from FAM, but works too with FAM. This is the idea: onde php daemon (file) that has two subfolder: 'parser' and 'plugins'. The folder 'parser' has the parser for different types of data and no dependency of libraries. Users can create a new parser just by creating a new file with an specific class name and some specific functions (that are called by main daemon process). Since daemon class 'scan' for new parser and plugins, user doesn't need to edit or insert any 'require' if they want to install a new plugin/parser: just drop files on corret folder and it's working! Plugins works the same way, but it's for processing data. All data coming from parser are sent to all active plugin. The plugins has a filename that consist of XXX-.class.php, where XXX are an integer number (any). This is used to make a priority system where 000 are called first and 999 last. Plugins can also change the data before send to another plugin......that's why we need a priority system.

This is the basic workflow: 1) Daemon create an object (ADSB) for each connection (must provide HOST, PORT and TYPE) 1.1) When object is created, they check TYPE to see if there's any parser for this TYPE 1.2) If there's an parser for this TYPE, a new object of 'parser' if created. The parser class if responsible for creating what they need (ex: sockets, cURL objects, etc).

2) A while call's a method called 'getData' for this ADSB (connection) object. 2.1) The ADSB object call's parser 'getData' to get data from device (or anything else) 2.2) The parser object return data from device/anything

3) The ADSB object get a list of plugins and order by priority (filename starting with 000) 3.1) The ADSB object create a new instance of each enabled plugin and send received data from parser. 3.2) Each plugin receive data and procces it....or, if needed, modify. 3.3) The next plugin in list will received the current data (even if they have beem modified by a previous plugin) and the 'original' data, if it needs.

All tis system is ready and it's very easy to use. I've made an parser for 'msg' type and i'm finishing a 'plugin' for inserting data into FAM.

The advantage of this system is: a) Don't need any library from FAM or anything else, just use PHP. b) It's very, very extendible and doesn't need to put a logic on just one file or library. If there's a new device, just create a new parser! c) It's easy to expand possibilities for the same data. For example: we can have one plugin to insert data on FAM; another plugin to check is aircraft has landed on destination airport or not; another plugin to check if aircraft if 'military' or anything else; d) Maintenance is easy. If you find any 'bug' on specific function (let's say, the code that checks for aircraft type), you don't need to find in a lot of code....just open this specific plugin.

Well, that's it. I hope I can help FAM to be even better with this system.

[Ysurac]

For now parsing system is easy so "parser" can be great and would be used. For plugins I'm not sure. But this part need some works, it's dirty.

What is needed:

• Data can be retrieved from multiple devices at the same time and same flight should be treated as same flight (this part is not easy even if possible via a plugin and a temporary table if multiple objects are used)
• All can be run as daemon or as cron job
• Need to be fast enough (it's not the case for now)
• Not too much replication of code
• Not too much CPU/memory

I hope to test your fork soon :)

[jmceara]

The main problem is 'fast enought' for now. My code doesn't do all the jobs at same time, it's sequencial, which is not so good. It's working, including plugins/parser, but I'm not happy with design. I'll start the same idea again, but using pthreads so we can make N jobs in paralell tasks without losing too much time. As soon as I have a prototype working, we talk again :)

[Ysurac]

I tried threads but I had far too much threads with many flights. I think it's the way to go but we can't open a thread by flight ;)

Good luck !

[jmceara]

Not by fly, but by source. Each source is one object (which is one thread). Using an object uses a little more CPU and memory then procedural, but it's more eleggant and expansible in the future (by extending to other functions/parsers/modules/etc..)

[Ysurac]

The problem is that we need to use same data for all sources (if the same flight is on multiple source). For this I use a memory table on mysql for the server version. But each thread/script is too slow. This take too much time with pics, airports, schedules,...

[jmceara]

It's because DB has a high IO. My idea is to use shared memory between sources and just in the end store on DB.

[jmceara]

Ycarus, the base is done. This is what I have for now: 1) Main object create one 'parser' object for each source. 2) Main object start each source to collect data 3) Each source must have a method that return cache collected data (an array with received flights) 4) The main object collect cached flight/data from each source and join then all in only one. This is where I'm thinking.....the join method. Do you think it's better to add a field to data array indicating the last received date/time? Ex: source 1 receive flight X and add to 'cached' array (on source object) and add to this data a field called 'received_on'. The source 2 receive the same flight, but some miliseconds after the source 1. Then, it add the same field, but with current date/time. I know that most sources have a field with received time......but if we implement a new source in the future that doesn't have? I must have one rule to decide which source has priority while receiving same data.

5) The received data are sent to 'plugins'. These plugins are responsible to insert/delete/change data on database and to clear flight data on main object. In this example, there would be at least two plugins: one for remove old data from flight data (in the main object) and another one to insert/edit data on database. Could be another one just to proccess some special functions, like 'arrivals'. This third plugin would analyze flight data and decide if a flight is an 'arrival' or 'departure'. The first plugin that I mentioned could be coded inside main object to remove old data, right? But I would like to make this the way that we could do almost anything in the future without need to change main code.

An least: for database operation, that's slow now, I'll use memory for some tables and after X seconds, dump to database. It's the fastest way. I'll contact you again to ask for DB operations of FAM.

[Ysurac]

Date/time from a lot of sources is not right (for ADS-B some programs use local time, some use UTC, some don't have a valid NTP server, some give strange data,...), so I use received time (at least it work for everybody).

Plugin need to check if data are correct before joining them. And all data don't need to be put in database (some flights send data too often).

For database this is an idea if it's really faster.

Good job, I really want to try :)

[jmceara]

That's what i tought. Every parser will input their 'received_time' into flight data before sending back to main object (that collects data from all sources). I'll try to make these checks tomorrow and send you some examples :)

[jmceara]

How much memory do you consider 'high'? Remember that today we have high processing power and high memory (even with small computers, like Raspberry Pi). I'm asking because I've made a simple test: create on php script that just fetch ALL data from routes table (which is the largest). Memory useage for this single test: 156,58Mb......plus some code for all other stuff. In my opinion, is not high, since with this all table will me in memory. Also, I'll try to find any memory-database (I think SQLite has this feature) that could use less memory.

[jmceara]

Another option is create a new thread just to do DB work. This way, the code that actually get data from sources doesn't get stucked by another proccess or DB I/O.

[Ysurac]

Web server, database and FlightAirMap should be usable on a Raspberry Pi 2 (1GB) with more than 3000 flights. MySQL/MariaDB can create Memory database/table. Route table can be populated on the fly with ACARS and airlines website data. I like the idea of a thread to do DB work.