performance of predicted/theoretical alarm

[ffxh]

I would like to ask you following, what are the outputs of predictive alarms and theoretical alarms and their comparisons in analyze_detection mode? In the process of running, I found that the evaluations used by these three outputs are empty. And for example, before getting the value of evaluation, the confusion matrix call returns to alarm_pred (which is a variable with a value of [1]) and then gets a matrix with a length of no more than one, so that I don't get some values in evaluation, resulting in an empty value returned by evaluation. When I'm trying to run this project I'm having a problem like this, can you help me understand or solve it ? Thank you so much!

[Ralphkaram10]

• I think that the reason why some values are not obtained in your evaluation is that the corresponding flights are too small: The code filters out flights whose number of meta messages (the rows of the csv) is smaller than the threshold alarm (via remove_small_flights_from_dict decorator function).
• A matrix with length of no more than one (1 item instead of 4) means you have only normal flights (label 0) or only attacked flights (label 1) in your dataset, in other words you only have one label type in your dataset.
• Which dataset did you use? Did you use one of those datasets provided by the link (they contain both types of labels: labels 0 and 1 so as to obtain a confusion matrix with 4 items) https://mega.nz/folder/xTcRFIxZ#hTj6vVoR-40ZgAx_9BSHmw?

[ffxh]

I used [callsign_AAL2434_filtered_post.csv], in the analyze_detection, my first output Evaluation of classification of meta messages: no problem, but the later outputs ran into a case where evaluation was empty.

[Ralphkaram10]

• The confusion matrix shown using the option --analyze_detection is by flight not by meta messages, in other words if a flight is considered normal, its label is 0 otherwise it is 1. And since you analyzed only one flight, you only have one label, therefeore the confusion matrix contains one item.
• As per the documentation, I advise you to test on a dataset from https://mega.nz/folder/xTcRFIxZ#hTj6vVoR-40ZgAx_9BSHmw to get a useful confusion matrix.
• Can you show me your config file and dataset used (According to your response, If I am not mistaken you used the dataset: dataset_example_containing_one_flight_only).

[ffxh]

Well, thanks for your help! I have solved the problem successfully with your help. Maybe I should add more flights, I tried to add another two flights and then I ran the code successfully. Now I'll try to open the link (accessing this site from where I am seems to time out) and use those datasets provided by it. Thank you very much for your response! Hope you are doing well!

[Ralphkaram10]

You are welcome, I’m glad to hear that you were able to solve the problem! If you have any more questions or need further assistance, feel free to ask! 😊

[ffxh]

Hello? Sorry to trouble you again, because the use reference URL in the download_sbs.py is now displayed as 404, is there any other reference method to run this code?

[Ralphkaram10]

Hello no worries

• You need an opensky-network account, and apply for data access on the website so that you are able to download adsb data using download_sbs.py

• The scripts below have the following roles: download_sbs.py: to download raw messages and turn them into the sbs format SBS_to_Meta.py: to turn sbs format into meta messages plot_flight.py: Plot the specified feature of a file of meta messages corresponding to one flight

• Note that download_sbs.py (which is based on the traffic library) is slow when downloading and processing flights, also some flights may not be downloadable. But it can be enough to get you to download a reasonable number of flights.

[ffxh]

Thank you very much for your help, sorry for the late reply, I have understood the way you provided, but because of some network access restrictions in my region, I have encountered the problem of access timeout, but I am constantly trying, I believe that I will get the data I need soon, and finally thank you again for your help, I hope you have a happy life. ^_^

[Ralphkaram10]

No worries, I am glad you understood the code. Good luck in getting the data. Hope you have a happy life too ^_^ !