logger and parser: development

[jerabaul29]

@jvoermans I think we are now quite close to having something that works.

• This platformio project:

https://github.com/jvoermans/Vibration_Logger/tree/master/material_Jean/Due_SD_high_frequency_logger

contains the high frequency logger code. I added the libraries as submodules in the lib folder, which is the "right" way to do it. This means that after pulling from github, you will need to execute a git submodule update --init to add all the necessary submodule files.

• This python code should take care of parsing the content of the raw SD card data:

https://github.com/jvoermans/Vibration_Logger/tree/master/material_Jean/BinarySdDataParser

• In addition, I have a couple of copies of scripts from the SdFat-beta project, that allow to test the SD card formatting etc (so this is just a copy of their code), may be a good idea to run before doing anything with the SD card:

https://github.com/jvoermans/Vibration_Logger/tree/master/material_Jean/ExFatFormatter

https://github.com/jvoermans/Vibration_Logger/tree/master/material_Jean/SdInfo

Can you:

• confirm that you are able to compile the logger code, after having issued the git submodule command
• check that you are able to run the code satisfactorily on your logger, and then to parse the obtained SD card data?

The logger uses a new file each 15 seconds now (will update to 15 minutes for "production"). It should log both the 5 first ADC channels, and the GPS, will add other sensors (or help you add them as I do not have them here :) ) in the days to come. The parser is still a bit simple, especially regarding parsing the GPS data, but I will develop this more in the days to come too. The GPS parsing also logs the PPS pin, so this should give an absolute time accuracy of typically a few microseconds, which should be great for synchronizing instruments.

I think this should be nearly ready when it regards the "pure logging" side. I think that for this time, may be best to not have iridium etc, but to rather have a very controlled deployment, and collect as much raw data as possible, so that we get some signal examples, and can learn about "what to look for in the signal and how to process it for sending Iridium metrics".

[jerabaul29]

PS: due to this line:

https://github.com/jvoermans/Vibration_Logger/blob/aaf8f9aa2f96643be69a84738d1b37c8a67f76db/material_Jean/Due_SD_high_frequency_logger/src/GPS_manager.h#L21

you may have a compiler error at first. To fix it, just change the value of the macro definition in the core (i.e. the location where the buffer size macro is defined) to 512. If you do not find how to do it, let me know in a separate issue, and I will help you fix that.

[jerabaul29]

FYI: I now added full support for the PPS GPS synchronization. I think it looks like I can get an accuracy of typically +- a few microseconds on absolute time, so I am very happy about this :) . This should really allow to track accurately where cracks take place if you can deploy several instruments at the same location.

Next week I will take some unpaid leave from my work at the Met office to work on several of my Univ projects. If you want to schedule a short zoom meeting to discuss this logger, let me know. I think it could be useful to have a short chat :) .

[jvoermans]

@jerabaul29

Got it all uploaded to the Due after some hickups (including change in buffer size). However, SD card is not working:

FastLogger enable serial debug begin() failed Do not reformat the SD. SdError: 0X18,0X0

Card was formatted as FAT32

Did you wire it as in the sketch? i.e. CS pin is digital 10? https://github.com/jvoermans/Vibration_Logger/blob/1eb060a5860222239aac30454b2185aefe689e38/material_Jean/Due_SD_high_frequency_logger/src/main.cpp#L12-L27

I double checked if it is the SD card or breakout. It works with the LowLatencyLogger, but only if I reduce the SPI Speed to 40 MHz... The SdInfo is working fine in reading the card. ExFatFormatter says card is too small. What SD card did you try, 16 GB?

[jerabaul29]

Mmh, this is very strange. What kind of SD card do you use? Can you double check the formatting into FAT32? My SD card is 32GB

And the wiring should be as in the comments:

Are you sure you issued the git submodules command and that the include point to SdFat-Beta, not another system wide installed SD library?

Very strange that the Formatter script does not work, it is taken straight from the library.

[jerabaul29]

Are you sure that you use the latest version of the git code? Can you try a git pull / check the status with git status?

[jvoermans]

Yes, had to git pull the whole thing this morning. And yes submodules are provided were installed.

Wiring looks the same. Maybe it is the sd card. But it is a regular SanDisk Ultra so not sure why though. I'll buy some other SD card tmrw/Saturday and try again!

[jerabaul29]

Do you have time for a quick chat / can you take the board and components home in order to be able to test on the next few days?

A few more questions I thought about:

• when you were successfully using the SD card a few days ago, it looks like you used another slave select pin. Can you try to wire to this pin instead, if necessary change the slave select pin here:

https://github.com/jvoermans/Vibration_Logger/blob/1eb060a5860222239aac30454b2185aefe689e38/material_Jean/Due_SD_high_frequency_logger/src/params.h#L48

Another thing is that maybe the SS macro is over written by some of your local libraries. Do you want to set it up "hard coded" to 4?

• which "kind" of Sd file / constructor were you using? In my case, given how I format the card, I need to use:

https://github.com/jvoermans/Vibration_Logger/blob/1eb060a5860222239aac30454b2185aefe689e38/material_Jean/Due_SD_high_frequency_logger/src/params.h#L38-L44

not sure if there may be some differences on windows.

[jerabaul29]

For some options around the sd fs and file:

https://github.com/greiman/SdFat-beta/blob/8adf5b5a86bab72c68a4fd6b32f45c4366c05209/examples/ExFatLogger/ExFatLogger.ino#L130-L144

Could you try the different options there?

[jvoermans]

Yes, used pin 53 before, but replacing SS with 53 didn't change anything. What do you mean with: "Another thing is that maybe the SS macro is over written by some of your local libraries. Do you want to set it up "hard coded" to 4?" ?

[jerabaul29]

Yes, exactly, can you try to hard code it to 4 and try with pin 4, and hard code with pin 53, and try 53?

Another thing: are you sure all your contacts are "good"? :) It easily happens to have a bad contact on breadboards, causing intermittent problems...

[jerabaul29]

Do you have time for a quick WhatsApp discussion?

[jerabaul29]

These problems are really strange. I do not understand why this happens. Do you think you would have the possibility to test this on a linux machine? :) If the SdFat-beta author wants to see the code, feel free to share my project with him of course :) .

[jerabaul29]

Maybe having a look at this can be illustrative of the challenges posed by different kinds of SD cards:

https://github.com/greiman/SdFat/issues/122

When you report the issue, can you say that it works with the old library, but not the new one? That may help him find the bug and fix it. (I think there is a bug somewhere in the newer library, or at least, that it does not handle non-conforming SD cards as well as the old library).

[jvoermans]

I just tried the example file ExFatLogger and get the same error. Using LowLatencyLogger example sketch works fine. I'll file an issue at greiman. You said you want to be linked to it or something? What do you mean/how do you do that?

[jerabaul29]

Sounds good.

If you can just tag me with @jerabaul29 that would be great :)

[jvoermans]

Just tagged, not sure if it is normal to do it at the bottom of the message ;)

Yes, I'll buy a different one, but the one I was using should be a real one. Maybe I've just been using it too much :)

[jerabaul29]

That's perfect!

Mmh, strange, we will see what he says.

[jerabaul29]

Which brand is it again?

[jerabaul29]

(I don't think the brand was included in your post there right?).

[jvoermans]

The ExFatLogger works when I change the SPI clock speed. Don't understand why though, it is a real SanDisk card. Can't imagine that that isn't good enough. I've been formatting and removing it a lot from the socket so maybe that is it.

[jerabaul29]

That is excellent. I will push the changes and let you know. You know, there is dark magics in SD cards ^^ .

[jerabaul29]

The limited frequency thing is pushed now. Can you try? :)

[jerabaul29]

If things go well you will see quite a bit of debug info during logging. The current file duration is 15 seconds. So you should wait for a couple of minutes, this should generate a few files, and then extract the SD card, and analyze the binary SD dumps with the code here:

https://github.com/jvoermans/Vibration_Logger/blob/master/material_Jean/BinarySdDataParser/BinaryParser.py

All you will need to do is adapt the path to your data, here:

https://github.com/jvoermans/Vibration_Logger/blob/f4bd79862a7603a262a3fb3a2a80dbeb244ebadd/material_Jean/BinarySdDataParser/BinaryParser.py#L398

The parser will be fully finished in a few days. I can also help you add logging for your other sensors etc in the c++ code :) .

[jvoermans]

Yes sketch is working now!

[jerabaul29]

Going to sleep, but that is excellent!

One thing: for the parser to work you need to use the PPS pin, and to have a GPS fix.

[jvoermans]

@jerabaul29 I tried to use the parser but run into an error. Is that because of the absence of the GPS fix?

Traceback (most recent call last): File "BinaryParser.py", line 400, in binary_folder_parser = BinaryFolderParser(folder=folder, show_plots=True) File "BinaryParser.py", line 221, in init binary_file_parser = BinaryFileParser(crrt_file, n_ADC_channels=self.n_ADC_channels) File "BinaryParser.py", line 49, in init self.parse_file() File "BinaryParser.py", line 54, in parse_file with open(self.path_to_file, "rb") as fh: TypeError: invalid file: PosixPath('Data_v1/F00000000.bin')

[jerabaul29]

Mmh no this is a path error. Are you sure the path is valid? Do you have the right number of 0s? Do you use the parser by providing a file or a folder / what exact code do you use?

[jvoermans]

I put a folder named 'Data_v1' with the data files inside in the folder where the parser is. I ran the file using an Ubuntu terminal for windows, cause the path naming differences between Linux and Windows causes a lot of errors.

I have to check what goes wrong with the name then...

[jerabaul29]

Ok. Can you post the code snippet you use to call the parser class? :)

[jvoermans]

Sorry, still beginner with Python. What do you mean exactly?

[jerabaul29]

No worries :)

Can you post your version of this:

https://github.com/jvoermans/Vibration_Logger/blob/f4bd79862a7603a262a3fb3a2a80dbeb244ebadd/material_Jean/BinarySdDataParser/BinaryParser.py#L398

And / or any other changes?

[jvoermans]

I only changed the path. The error I got above comes from: folder = Path("./Data_v1/")

If I do: folder = Path("/Data_v1/")

gives: Traceback (most recent call last): File "BinaryParser.py", line 400, in binary_folder_parser = BinaryFolderParser(folder=folder, show_plots=True) File "BinaryParser.py", line 211, in init previous_filenumber = filename_to_filenumber(self.list_files[0]) IndexError: list index out of range

[jerabaul29]

Ok. Dont do:

folder = Path("/Data_v1/")

Because that would ask the parser to look for data at the root of your machine. I expect this to fail, so no surprises here.

The right way to do it is as you wrote:

folder = Path("./Data_v1/")

I am not sure why it does not work, You are on windows right? ^^ This is probably the explanation somehow, but not sure why...

A few things:

• are you able to run my code as is on the master branch of the repo, ie against my test data, or do you get the same kind of error?
• Can you check that there is well a file Data_v1/F00000000.bin that is valid, that has non 0 size, etc?
• Do you have some ways to try on linux?

[jerabaul29]

(as previously discussed, if you can convince your department to buy you a linux laptop, that would be great. Will save you and me a lot of time on the long run :) )

[jvoermans]

It is a Windows problem. I made a Bootable key with Linux and tested the example folder and works fine. Unfortunately I can't use the key on my work computer and my personal computer is too old to be used for work.

And all computers used for work are controlled by UoM, so there is not chance I'll be able to arrange a second computer. They are pretty strict on that.

I tried the example folder in the Ubuntu terminal, but gives the same error:

joeyvoermans@4320L-142139-W:/mnt/c/Users/jvoermans/OneDrive - The University of Melbourne/Documents/GitHub/Vibration_Logger/material_Jean/BinarySdDataParser$ python3 BinaryParser.py

Traceback (most recent call last): File "BinaryParser.py", line 400, in binary_folder_parser = BinaryFolderParser(folder=folder, show_plots=True) File "BinaryParser.py", line 221, in init binary_file_parser = BinaryFileParser(crrt_file, n_ADC_channels=self.n_ADC_channels) File "BinaryParser.py", line 49, in init self.parse_file() File "BinaryParser.py", line 54, in parse_file with open(self.path_to_file, "rb") as fh: TypeError: invalid file: PosixPath('example_data/F00000000.bin')

[jerabaul29]

Mmmh, which python version do you use? See this issue:

https://stackoverflow.com/questions/42694112/when-using-pathlib-getting-error-typeerror-invalid-file-posixpathexample-t

The fix may be to write:

with open(str(self.path_to_file), "rb") as fh:

Instead. Can you try?

[jerabaul29]

That may actually be an 'old python3 version' problem and not a Windows problem ^^ .

[jvoermans]

Interesting, I was looking at same website but didn't understand it ;)

Still get error:

Traceback (most recent call last): File "BinaryParser.py", line 400, in binary_folder_parser = BinaryFolderParser(folder=folder, show_plots=True) File "BinaryParser.py", line 245, in init self._generate_utc_timestamps_regression(show_fit=self.show_plots) File "BinaryParser.py", line 312, in _generate_utc_timestamps_regression ) for crrt_entry in self.dict_data["PPS_GPRMC_entries"] File "BinaryParser.py", line 312, in ) for crrt_entry in self.dict_data["PPS_GPRMC_entries"] TypeError: combine() takes at most 2 arguments (3 given)

I use Python 3.5.2

[jerabaul29]

Ok. Can you try to update to the latest python version?

[jerabaul29]

The new error is probably also due to old python3 version.

[jvoermans]

Ok stupid question, how to I update this easily?

[jvoermans]

Ok ignore the previous comment. I installed python3.7.9, now the script runs on your example files.

Sorry, I'm overly relying on your expertise here, but perhaps you know. I've got Spyder interface to work with python. Any idea why running the script in Spyder gives me 2 figures, while running it in the Ubuntu Terminal runs it without error but also without giving figures?

Running the data files I obtained (those without GPS connection) give an error though, but maybe that's because there wasn't a connection....

Traceback (most recent call last): File "BinaryParser.py", line 400, in binary_folder_parser = BinaryFolderParser(folder=folder, show_plots=True) File "BinaryParser.py", line 242, in init self._unwrap_all_micros() File "BinaryParser.py", line 301, in _unwrap_all_micros self.dict_data["PPS_entries_micros_unwrapped"] = unwrapp_list_micros(self.dict_data["PPS_entries_micros"]) File "BinaryParser.py", line 166, in unwrapp_list_micros previous_entry_unwrapped = list_micros[0] IndexError: list index out of range

[jerabaul29]

Can the missing figure be a backend problem on windows?

https://codeyarns.com/tech/2016-03-14-matplotlib-plot-is-not-displayed-in-window.html

Yes, in case of no GPS fix I expect things to not work. I will push an update to make things work also without GPS fix today :) .

[jerabaul29]

Now cases when no GPS information is available should work. Of course the code cannot know the date, so will be offsetted to 1970. Can you confirm this works on your machine?

[jerabaul29]

@jvoermans the parser should be fully ready now. I have added a small explanatory script:

https://github.com/jvoermans/Vibration_Logger/blob/master/material_Jean/BinarySdDataParser/script_perform_parsing.py

Can you:

• check that you are able to fully execute the script
• check that you are able to use it on your own data

Once this is ok, I consider development of the parser and most of the logger ready.

[jvoermans]

Sorry for delay, was busy with proposal. Should have more time Thursday and Friday.

Thanks, it works on your data examples and I can see that variables are created. It doesn't work on my data though. I tried to upload them but accidently placed them all in the main folder....

What are the pkl files?

[jerabaul29]

No worries.

Ok, I will move these to a separate folder.

the pkl files are serialized dumps of the processed data. Basically, once a file is parsed, I dump it as pkl, and the pkl can be read natively by python to get the parsed data. This way we do not need to use a lot of time parsing the files at the beginning of each script, we can just load "pre parsed" data. And pkl is more efficient with space than the csv I was using in the IMU logger, which is good if we await 10s or 100s of GB of data.

[jerabaul29]

There were a couple of problems triggered by not having PPS info. This should be fixed now. I am able to parse your file.

If you have any further problems parsing any data just let me know :) .

[jerabaul29]

Closing this issue. If you have some specific problems, just open new specific issues :) .