Feature Request: Saving the output

[Steffen-W]

Hi @josevcm,

Your project is awesome and I love using it. I would be very happy if there was a way to save the output as JSON, CSV or whatever. So not only the frame but everything that is displayed in the table (time, delta, rate, type, event and frame). That would be really great.

[josevcm]

Hi Steffen

Thanks for your comment!

You can use the "save" button to write a JSON file with details of the captured data. I have documented the format in the README.md file.

Please check if this meets your requirements.

Regards

[Steffen-W]

Hi @josevcm,

I also discovered it the day after. I personally didn't find it intuitive, but that's not so bad. Your code is great and extremely helpful!

In a shortened form I have inserted python code with which the json file can be read in. I need it for further analysis. A lot is already done in nfc-laboratory but I find the packages themselves easier to analyze in python.

import pandas as pd
from enum import Enum
import glob

jpgFilenamesList = glob.glob("record-*.json")
file_path = jpgFilenamesList[0]
print("Read", file_path)

class TechType(Enum):
    none = 0
    NfcA = 1
    NfcB = 2
    NfcF = 3
    NfcV = 4

class RateType(Enum):
    r106k = 0
    r212k = 1
    r424k = 2
    r848k = 3

class FrameType(Enum):
    CarrierOff = 0
    CarrierOn = 1
    PollFrame = 2
    ListenFrame = 3

class FrameFlags(Enum):
    ShortFrame = int("0x01", 16)
    Encrypted = int("0x02", 16)
    Truncated = int("0x08", 16)
    ParityError = int("0x10", 16)
    CrcError = int("0x20", 16)
    SyncError = int("0x40", 16)

class FramePhase(Enum):
    CarrierFrame = 0
    SelectionFrame = 1
    ApplicationFrame = 2

class ISO15693Command(Enum):
    Inventory = int("0x01", 16)
    StayQuiet = int("0x02", 16)
    ReadSingleBlock = int("0x20", 16)
    WriteSingleBlock = int("0x21", 16)
    LockBlock = int("0x22", 16)
    ReadMultiple = int("0x23", 16)
    WriteMultipleBlocks = int("0x24", 16)
    Select = int("0x25", 16)
    ResettoReady = int("0x26", 16)
    WriteAFI = int("0x27", 16)
    LockAFI = int("0x28", 16)
    WriteDSFID = int("0x29", 16)
    LockDSFID = int("0x2A", 16)
    GetSystemInformation = int("0x2B", 16)
    GetMultipleBlockSecurityStatus = int("0x2C", 16)

class FrameData:
    def __init__(self, flag, data, crc):
        self.flag = flag
        self.data = data
        self.crc = crc

    def __repr__(self):
        return f"FrameData(flag={self.flag}, data={self.data}, crc={self.crc})"

    def __str__(self):
        flag_str = f"Flag: {hex(self.flag)}" if self.flag is not None else "Flag: None"
        crc_str = (
            f"CRC: {[hex(x) for x in self.crc]}"
            if self.crc is not None
            else "CRC: None"
        )
        data_str = f"Data: {[hex(x) for x in self.data]}" if self.data else "Data: None"
        return f"{flag_str}, {data_str}, {crc_str}"

# Convert Enum in to DataFrame
def translate_enum(enum_class, value):
    try:
        return enum_class(value).name
    except ValueError:
        return value

def parse_frame_data(frame_data_str):
    hex_values = frame_data_str.split(":")
    int_values = [int(x, 16) for x in hex_values]
    if len(int_values) > 3:
        flag = int_values[0]
        crc = int_values[-2:]
        data = int_values[1:-2]
    elif len(int_values) == 3:
        flag = int_values[0]
        crc = int_values[-2:]
        data = []
    else:
        flag = None
        crc = int_values
        data = []
    return FrameData(flag, data, crc)

def convert_to_hex(obj):
    if isinstance(obj, dict):
        return {k: convert_to_hex(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [convert_to_hex(i) for i in obj]
    elif isinstance(obj, int):
        return "0x{:02x}".format(obj)
    else:
        return obj

# Read JSON
df = pd.read_json(file_path)

df = df["frames"]
df = pd.json_normalize(df)
# "frameData", "frameFlags", "framePhase", "frameRate", "frameType", "sampleEnd", "sampleStart", "techType", "timeEnd", "timeStart"

df["techType"] = df["techType"].apply(lambda x: translate_enum(TechType, x))
df["frameType"] = df["frameType"].apply(lambda x: translate_enum(FrameType, x))
df["framePhase"] = df["framePhase"].apply(lambda x: translate_enum(FramePhase, x))
df["frameData"] = df["frameData"].apply(parse_frame_data)
df["frameFlags"] = df["frameFlags"].apply(
    lambda x: [flag.name for flag in FrameFlags if x & flag.value]
)

for index, row in df.iterrows():
    print(
        "{:02.5f}".format(row["timeStart"]),
        convert_to_hex(row["frameData"].flag),
        convert_to_hex(row["frameData"].data),
    )

[josevcm]

Great job! I'm not an expert in python but it seems like a very powerful language to me. Thank you for your comments, if you need anything else let me know.

[Steffen-W]

Hi @josevcm,

It's a different issue but goes in the same direction. Would it be possible to output received packets directly in the terminal so that I can analyze them live? Preferably via a debug flag at the beginning. I have the problem that the transmission from the device to be analyzed rarely causes errors and I can't tell straight away whether everything is working correctly or not. I would really like to have the packets analyzed live in a separate script and be informed promptly.

I can also do this myself, but can you give me a tip on the best way to do this?

Thank you very much. Your code works really well!

[josevcm]

Hi Steffen

In the repository there is already a tool to perform regression tests that works on the command line, it could be modified very easily for your needs, you have it in /src/nfc-test/app-test/src/main/cpp/main.cpp , now it is launched giving the path to the "wav" folder in the repository as a parameter but it would be easy to adapt it. Tell me if it's worth it, or if you give me a few days I can do it myself.

[Steffen-W]

Hi,

i've been looking at your code for the last few days but it looks really good throughout! As you have already written, /src/nfc-test/app-test/src/main/cpp/main.cpp is for .wav files and is very clearly written. I would prefer to record the data live, but I think you understood that anyway. I would be delighted if the code could be extended in this way. However, I thought that it would be easier for programming to additionally output the received packets in the terminal when the gui is running. I would then read them in continuously and evaluate them. I would of course make the code available for analysis so that others could also benefit from it. I could imagine that not only I would benefit from this. It's just easier to debug if you don't have to look very closely all the time. :)

It will be a python solution in my case, as you can probably already guess.

[josevcm]

Ok, give me some time and I will add this feature to testing tool, is really easy, may be today

[Steffen-W]

Thank you very much

[josevcm]

The nfc-rx tool is ready in src/nfc-test/app-rx, can you check it? Hope this can help you

Usage: [-v] [-d] [-p nfca,nfcb,nfcf,nfcv] [-t nsecs] v: verbose mode, write logging information to stderr d: debug mode, write WAV file with raw decoding signals (highly affected performance!) p: enable protocols, by default all are enabled t: stop capture after number of seconds

[josevcm]

example output

nfc-rx.exe 000000.000 (CarrierOff) 000003.704 (CarrierOn) 000003.704 (CarrierOff) 000003.724 (CarrierOn) 000003.734 (PCD->PICC) [NfcA@106]: C2 E0 B4 000003.749 (PCD->PICC) [NfcA@106]: 52 000003.749 (PICC->PCD) [NfcA@106]: 44 03 000003.750 (PCD->PICC) [NfcA@106]: 93 70 88 04 4A 6A AC 4A F9 000003.751 (PICC->PCD) [NfcA@106]: 24 D8 36 000003.752 (PCD->PICC) [NfcA@106]: 95 70 32 77 46 80 83 9C A4 000003.753 (PICC->PCD) [NfcA@106]: 20 FC 70 000003.753 (PCD->PICC) [NfcA@106]: E0 80 31 73 000003.754 (PICC->PCD) [NfcA@106]: 06 75 77 81 02 80 02 F0 000003.776 (PCD->PICC) [NfcA@106]: 02 90 5A 00 00 03 00 00 00 00 61 28 000003.779 (PICC->PCD) [NfcA@106]: 02 91 00 29 10 000003.791 (PCD->PICC) [NfcA@106]: 03 90 60 00 00 00 3F 9C 000003.792 (PICC->PCD) [NfcA@106]: 03 04 01 01 01 00 18 05 91 AF 8B 5D 000003.799 (PCD->PICC) [NfcA@106]: C2 E0 B4 000003.799 (PICC->PCD) [NfcA@106]: C2 E0 B4 000003.801 (PCD->PICC) [NfcA@106]: 52

[Steffen-W]

Unfortunately, I do not have the file "src/nfc-test/app-rx" after compiling.

OS: Linux Mint 21.3 x86_64

[Steffen-W]

Add #include <csignal> to src/nfc-test/app-rx/src/main/cpp/main.cpp#L30 otherwise compile does not work in nfc-laboratory_test/cmake-build-release/src/nfc-test for me.

[josevcm]

Sorry I only tried it on Windows, tomorrow if I have time I will review the Build for Linux

[Steffen-W]

I had already thought that. Fortunately, I also have a Windows and a Mac at my disposal, but I had only tested it under Linux.

with an nfcv tag

./cmake-build-release/src/nfc-test/app-rx/nfc-rx -p nfcv -t 10
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Disabled direct sampling mode
Invalid sample rate: 10000000 Hz
000000.000 (CarrierOff) 
000000.108 (CarrierOn) 
rtlsdr_demod_write_reg failed with -9
r82xx_write: i2c wr failed=-9 reg=06 len=1

with an nfca tag

./cmake-build-release/src/nfc-test/app-rx/nfc-rx
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Disabled direct sampling mode
Invalid sample rate: 10000000 Hz
000000.000 (CarrierOff) 
000001.165 (CarrierOn) 
000001.168 (CarrierOff) 
000001.179 (CarrierOn) 
000001.179 (CarrierOff) 
000003.156 (CarrierOn) 
000003.185 (CarrierOff) 
000003.523 (CarrierOn) 
000003.548 (PCD->PICC) [NfcA@424]: 2A 
000003.560 (PCD->PICC) [NfcA@424]: 2A 
000003.599 (PCD->PICC) [NfcA@424]: 16 
000003.601 (PCD->PICC) [NfcA@424]: 2A 
000003.615 (PCD->PICC) [NfcA@424]: 2A 

Certainly not an ideal picture, but more than in the terminal. I can test it later under linux.

It would be great if you could have a look at it under Linux tomorrow.

I would also like to point out that when the gui is called up by default, the terminal is described diligently. This is certainly irritating for some people. If you could switch this on and off via flags, it would certainly be great.

[josevcm]

I see that you are using an rtlsdr, in that case I have to adapt it because I had prepared it to use it with airspy, tomorrow I will try to add support for rtl and Building on Linux, sorry

[Steffen-W]

I'll try to reproduce the code or make the necessary changes myself.

Logger.h#L54 WARN_LEVEL is probably more suitable as a standard.

main.cpp#L172 should probably also pass the debug level.

One possibility would be the following change in StreamModel.cpp#L451-L481. So at first glance it does what it is supposed to do, but it is really extremely unclean. 😅

   if (role == Qt::DisplayRole || role == Qt::UserRole)
   {
      QString result;
      switch (index.column())
      {
         case Columns::Id:
            result = index.row();
            return result;
         case Columns::Time:
            result = impl->frameTime(frame);
            printf("{frameTime: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Delta:
            result = impl->frameDelta(frame, prev);
            printf("frameDelta: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Rate:
            result = impl->frameRate(frame);
            printf("frameRate: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Tech:
            result = impl->frameTech(frame);
            printf("frameTech: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Event:
            result = impl->frameEvent(frame, prev);
            printf("frameEvent: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Flags:
            result = impl->frameFlags(frame);
            printf("frameFlags: %s, ", result.toStdString().c_str());
            return result;
         case Columns::Data:
            result = impl->frameData(frame);
            printf("Frame Data: %s}\n", result.toStdString().c_str());
            return result;
      }

      return {};
   }

[josevcm]

Debug console is only displayed if build is done in "development mode", to remove console and write logging to file, you must build with -DBUILD_PROJECT_VERSION=1.0.0 for example.

Now... I have already fix issues and tested in linux and check than works with RTLSDR:

[Steffen-W]

I still had to compile the code with make, when compiling the whole code it was not created. I hope this is correct. Unfortunately it still doesn't work for me.

$ ./cmake-build-release/src/nfc-test/app-rx/nfc-rx 
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Disabled direct sampling mode
000000.000 (CarrierOff) 
rtlsdr_demod_write_reg failed with -9
r82xx_write: i2c wr failed=-9 reg=05 len=1

[josevcm]

It's strange, with the QT interface it works?

[josevcm]

If you want to obtain very good results, I recommend this receiver, it is the best I have tried for its price: https://itead.cc/product/airspy-mini/

[Steffen-W]

Hi, I've thought about that too. Too bad I hadn't ordered it yet. But I will today.

Unfortunately, I don't know exactly what you mean if it works with the QI interface. But the GUI works very well and I am satisfied.

Can you tell me where the following function is called? I somehow couldn't figure it out clearly. StreamModel.cpp

QVariant StreamModel::data(const QModelIndex &index, int role) const

[josevcm]

Qt is the name of the graphical interface, that's what I meant.

That function is automatically invoked by the Qt framework when it needs to access the data to paint it in the graphical interface table.

Theoretically the nfc-rx executable uses the same processes to capture and analyze the signals but without dependencies on graphical libraries, it should work the same.

[Steffen-W]

git checkout 136df88f83e76b22b53719b8f

cmake -DCMAKE_BUILD_TYPE=Release -S nfc-laboratory -B cmake-build-release
cmake --build cmake-build-release --target nfc-lab -- -j 6
cp nfc-laboratory/dat/config/nfc-lab.conf .
./cmake-build-release/src/nfc-app/app-qt/nfc-lab

git checkout master 

cmake -DCMAKE_BUILD_TYPE=Release -S nfc-laboratory -B cmake-build-release
cmake --build cmake-build-release --target nfc-lab -- -j 6
cp nfc-laboratory/dat/config/nfc-lab.conf .
./cmake-build-release/src/nfc-app/app-qt/nfc-lab

git checkout master

cmake -DCMAKE_BUILD_TYPE=Debug -S nfc-laboratory -B cmake-build-debug cmake --build cmake-build-debug --target nfc-lab -- -j 6 cp nfc-laboratory/dat/config/nfc-lab.conf . ./cmake-build-debug/src/nfc-app/app-qt/nfc-lab

$ cd cmake-build-debug/src/nfc-test/app-rx
$ make
$ ./nfc-rx 
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Disabled direct sampling mode
000000.000 (CarrierOff) 
rtlsdr_demod_write_reg failed with -9
r82xx_write: i2c wr failed=-9 reg=05 len=1

[Steffen-W]

diff --git a/src/nfc-app/app-qt/src/main/cpp/model/StreamModel.cpp b/src/nfc-app/app-qt/src/main/cpp/model/StreamModel.cpp
index 671ae1b..ce12835 100644
--- a/src/nfc-app/app-qt/src/main/cpp/model/StreamModel.cpp
+++ b/src/nfc-app/app-qt/src/main/cpp/model/StreamModel.cpp
@@ -450,31 +450,47 @@ QVariant StreamModel::data(const QModelIndex &index, int role) const

    if (role == Qt::DisplayRole || role == Qt::UserRole)
    {
+      QString result;
       switch (index.column())
       {
          case Columns::Id:
-            return index.row();
+            result = index.row();
+            return result;

          case Columns::Time:
-            return impl->frameTime(frame);
+            result = impl->frameTime(frame);
+            printf("{frameTime: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Delta:
-            return impl->frameDelta(frame, prev);
+            result = impl->frameDelta(frame, prev);
+            printf("frameDelta: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Rate:
-            return impl->frameRate(frame);
+            result = impl->frameRate(frame);
+            printf("frameRate: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Tech:
-            return impl->frameTech(frame);
+            result = impl->frameTech(frame);
+            printf("frameTech: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Event:
-            return impl->frameEvent(frame, prev);
+            result = impl->frameEvent(frame, prev);
+            printf("frameEvent: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Flags:
-            return impl->frameFlags(frame);
+            result = impl->frameFlags(frame);
+            printf("frameFlags: %s, ", result.toStdString().c_str());
+            return result;

          case Columns::Data:
-            return impl->frameData(frame);
+            result = impl->frameData(frame);
+            printf("frameData: %s}\n", result.toStdString().c_str());
+            return result;
       }

       return {};
diff --git a/src/nfc-lib/lib-rt/rt-lang/src/main/include/rt/Logger.h b/src/nfc-lib/lib-rt/rt-lang/src/main/include/rt/Logger.h
index a45110d..2203ef4 100644
--- a/src/nfc-lib/lib-rt/rt-lang/src/main/include/rt/Logger.h
+++ b/src/nfc-lib/lib-rt/rt-lang/src/main/include/rt/Logger.h
@@ -51,7 +51,7 @@ class Logger
          TRACE_LEVEL = 5
       };

-      explicit Logger(const std::string &name, int level = INFO_LEVEL);
+      explicit Logger(const std::string &name, int level = WARN_LEVEL);

       void trace(const std::string &format, std::vector<Variant> params = {}) const;

cmake -DCMAKE_BUILD_TYPE=Release -S nfc-laboratory -B cmake-build-modified cmake --build cmake-build-modified --target nfc-lab -- -j 6

For "./cmake-build-modified/src/nfc-test/app-rx/nfc-rx", my botch-up doesn't help me either. Even if it works amazingly well with the gui. I've ordered the SDR, it will take a few days and I'm curious to see how it compares.

[josevcm]

I see that you are already an expert!