search

cyberman54 / ESP32-Paxcounter

Wifi & BLE driven passenger flow metering with cheap ESP32 boards
https://cyberman54.github.io/ESP32-Paxcounter/
Other
1.73k stars 405 forks source link

[Feature request] use SD card to store Timestamp+Counter. #500

Closed ajpaezm closed 4 years ago

ajpaezm commented 4 years ago

Hi,

I'd like to know if anyone here managed to use the SD card successfully to store anything from the Paxcounter.

What I want to do is simple, I want to have a back-up from the counters and timestamps for posterior analysis.

These are the contents of sdcard.h and sdcard.cpp I've written, using the libraries for the SD card of ESP32, my board is a LoRa32 V2.1_1.6:

sdcard.h:

#ifndef _SDCARD_H
#define _SDCARD_H

#include "globals.h"

#include "FS.h"
#include "SD.h"
#include "SPI.h"

void listDir(fs::FS &fs, const char * dirname, uint8_t levels);
void writeFile(fs::FS &fs, const char * path, const char * message);
void appendFile(fs::FS &fs, const char * path, const char * message);

int sd_init(void);
int save_scan(time_t timestamp);

#endif // _SDCARD_H

sdcard.cpp:

#include "sdcard.h"

int sdStatus = 1;

void listDir(fs::FS &fs, const char * dirname, uint8_t levels){
    Serial.printf("Listing directory: %s\n", dirname);

    File root = fs.open(dirname);
    if(!root){
        Serial.println("Failed to open directory");
        return;
    }
    if(!root.isDirectory()){
        Serial.println("Not a directory");
        return;
    }

    File file = root.openNextFile();
    while(file){
        if(file.isDirectory()){
            Serial.print("  DIR : ");
            Serial.println(file.name());
            if(levels){
                listDir(fs, file.name(), levels -1);
            }
        } else {
            Serial.print("  FILE: ");
            Serial.print(file.name());
            Serial.print("  SIZE: ");
            Serial.println(file.size());
        }
        file = root.openNextFile();
    }
}

void writeFile(fs::FS &fs, const char * path, const char * message){
    Serial.printf("Writing file: %s\n", path);

    File file = fs.open(path, FILE_WRITE);
    if(!file){
        Serial.println("Failed to open file for writing");
        return;
    }
    if(file.print(message)){
        Serial.println("File written");
    } else {
        Serial.println("Write failed");
    }
    file.close();
}

void appendFile(fs::FS &fs, const char * path, const char * message){
    Serial.printf("Appending to file: %s\n", path);

    File file = fs.open(path, FILE_APPEND);
    if(!file){
        Serial.println("Failed to open file for appending");
        return;
    }
    if(file.print(message)){
        Serial.println("Message appended");
    } else {
        Serial.println("Append failed");
    }
    file.close();
}

int sd_init() {
  SPIClass spi(VSPI);
  spi.begin(SD_SCK, SD_MISO, SD_MOSI, SD_CS);
  if (!SD.begin(SD_CS, spi)) {
    ESP_LOGI(TAG, "Error initializing SD card");
    return -1;
  }
  // checking for SD card type:
  uint8_t cardType = SD.cardType();
  if(cardType == CARD_NONE)
  {
    ESP_LOGI(TAG, "The SD card is not properly attached to the slot");
  }
  if (cardType == CARD_MMC)
  {
    ESP_LOGI(TAG, "SD card type: MMC");
  }
  else if (cardType == CARD_SD)
  {
    ESP_LOGI(TAG, "SD card type: SD");
  }
  else if (cardType == CARD_SDHC)
  {
    ESP_LOGI(TAG, "SD card type: SDHC");
  }
  else if (cardType == CARD_UNKNOWN)
  {
    ESP_LOGI(TAG, "SD card type: Unknown.");
  }
  // getting size of the SD card:
  uint64_t cardSize = SD.cardSize()/(1024*1024);
  ESP_LOGI(TAG, "Size %d", cardSize);
  //creating a directory to fill:
  listDir(SD, "/", 0);
  writeFile(SD, "/myCounters.txt", "Counter values:");

  ESP_LOGI(TAG, "SD card initialized");
  sdStatus = 0;
  return 0;
}

int save_scan(time_t timestamp)
{
    char filename[128];
    sprintf(filename, "/%lu.scn", timestamp);
    ESP_LOGI(TAG, "Flag 1");
    fs::File file = SD.open(filename, FILE_WRITE);
    ESP_LOGI(TAG, "Flag 2");
    if(file == NULL)
    {
        ESP_LOGI(TAG, "Could not open file %s", filename);
        return -1;
    }
    file.printf("%X", macs_wifi);
    file.close();
    return 0;
}

Is all good when I initialize it or even create the test file "myCounters.txt", but then, when I try to call "save_scan" or "writeFile" or "appendFile" inside senddata.cpp, I get this error (after reaching the same line of code for a second time, the first time what it does is rebooting):

assertion "res == coreID || res == portMUX_FREE_VAL" failed: file "/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.inc.h", line 105, function: vPortCPUAcquireMutexIntsDisabledInternal abort() was called at PC 0x4011164f on core 1

Backtrace: 0x400929d8:0x3ffdda00 0x40092c09:0x3ffdda20 0x4011164f:0x3ffdda40 0x4008fb05:0x3ffdda70 0x4008e7ac:0x3ffddaa0 0x400e0759:0x3ffddae0 0x400d6aa9:0x3ffddb00 0x400dcd23:0x3ffddb30 0x400dd255:0x3ffddb60 0x400f51ca:0x3ffddba0 0x400f1995:0x3ffddbc0 0x400f2cd2:0x3ffddbe0 0x400f2e21:0x3ffddc00 0x400f3cb2:0x3ffddc30 0x400f4df0:0x3ffddea0 0x4010fae5:0x3ffde010 0x4000bcc5:0x3ffde030 0x4015ab25:0x3ffde050 0x401402e9:0x3ffde070 0x400dcaa8:0x3ffde0f0 0x400d593d:0x3ffde120 0x400d5dc9:0x3ffde160 0x400d2f47:0x3ffde1a0 0x4008eadd:0x3ffde1d0

Rebooting...

[E][vfs_api.cpp:22] open(): File system is not mounted

Are there any suggestions to this on how to solve this? Is this an issue about the core being used for the task? I'm lost to be fair.

Any assistance on this I'd be extremely glad.

cyberman54 commented 4 years ago

I read about issues with the SD card logic on some chinese TTGO boards, which needs some special treatment.

You may try asking this forum.

Since this is not a paxcounter related issue, i close this issue. But you're welcome to add comments.

cyberman54 commented 4 years ago

Reopened labelled as feature enhancement.

Nethranv commented 4 years ago

Yeah too much printing in memory is causing this rebooting.. and SD card is not closed properly so it cant open again..

Try keeping your ESP_LOGI to minimum data and let me know if it works.. we too are doing similar stuff.

AugustQu commented 4 years ago

Hi,

I just did it.

Here is the output of my code:

date, time, wifi, bluet 00.00.1970,00:01:11,5,0 00.00.1970,00:02:11,1,0 00.00.1970,00:03:11,3,0

this is just my first test. The format is CSV.

I will have to clean up my code, add some comments etc. Then I will upload it here.

AugustQu commented 4 years ago

this is the board I used: https://www.aliexpress.com/item/32990008126.html

The board already includes the SDcard-reader.

And I also have no access to a lorawan-gateway here in my little town. So I needed this.

I think I used this library: https://github.com/nhatuan84/esp32-micro-sdcard/blob/master/mySD.h

cyberman54 commented 4 years ago

Thanks! If you commit your Pull Request, please use development branch as base, not master. Thanks.

AugustQu commented 4 years ago

I will do it but please give me some time.

AugustQu commented 4 years ago

In the moment I can't do it because I will go to a vacation. When I come back I will work on this topic.

ajpaezm commented 4 years ago

Hey guys, sorry I didn't stay on the loop of this.

Hi @Nethranv! You said:

Yeah too much printing in memory is causing this rebooting.. and SD card is not closed properly so it cant open again..

Try keeping your ESP_LOGI to minimum data and let me know if it works.. we too are doing similar stuff.

I tried this and didn't had much luck :(

Hi @AugustQu! Thanks so much for the interest in this one. I'm curious and looking forward about what you did to make it work.

Hi,

I just did it.

Here is the output of my code:

date, time, wifi, bluet 00.00.1970,00:01:11,5,0 00.00.1970,00:02:11,1,0 00.00.1970,00:03:11,3,0

this is just my first test. The format is CSV.

I will have to clean up my code, add some comments etc. Then I will upload it here.

cyberman54 commented 4 years ago

@ajpaezm when using SPI interface of card reader, have in mind that paxcounter is using SPI for communication with the LORA chip. (Only) If both use same SPI hardware interface you need a mutex when accessing the SPI bus, otherwise there will be race conditions on multitasking esp32 platform.

The above mentioned TTGO board is using separate interface for SD card reader, so hardware should be suitable.

SD_DATA1 - GPIO4
SD_DATA0 - GPIO2
SD_CLK - GPIO14
SD_CMD - GPIO15
SD_DATA3 - GPIO13
SD_DATA2 - GPIO12
Nethranv commented 4 years ago

WE are able to save the MAC address , timestamp, RSSI into SD card. I will share the code for files i modified.

Nethranv commented 4 years ago

Main.cpp: ESP_LOGE(TAG, "ZzZzZzZz **** ZZzZzZzZzZ***"); ESP_LOGI(TAG, "Using SDMMC peripheral"); sdmmc_host_t host = SDMMC_HOST_DEFAULT();

// This initializes the slot without card detect (CD) and write protect (WP) signals.
// Modify slot_config.gpio_cd and slot_config.gpio_wp if your board has these signals.
sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();

// To use 1-line SD mode, uncomment the following line:
 slot_config.width = 1;

// GPIOs 15, 2, 4, 12, 13 should have external 10k pull-ups.
// Internal pull-ups are not sufficient. However, enabling internal pull-ups
// does make a difference some boards, so we do that here.
gpio_set_pull_mode(GPIO_NUM_15, GPIO_PULLUP_ONLY);   // CMD, needed in 4- and 1- line modes
gpio_set_pull_mode(GPIO_NUM_2, GPIO_PULLUP_ONLY);    // D0, needed in 4- and 1-line modes
gpio_set_pull_mode(GPIO_NUM_4, GPIO_PULLUP_ONLY);    // D1, needed in 4-line mode only
gpio_set_pull_mode(GPIO_NUM_12, GPIO_PULLUP_ONLY);   // D2, needed in 4-line mode only
gpio_set_pull_mode(GPIO_NUM_13, GPIO_PULLUP_ONLY);   // D3, needed in 4- and 1-line modes

// Options for mounting the filesystem.
// If format_if_mount_failed is set to true, SD card will be partitioned and
// formatted in case when mounting fails.
esp_vfs_fat_sdmmc_mount_config_t mount_config = {
    .format_if_mount_failed = true,
    .max_files = 5,
    .allocation_unit_size = 16 * 1024
};

// Use settings defined above to initialize SD card and mount FAT filesystem.
// Note: esp_vfs_fat_sdmmc_mount is an all-in-one convenience function.
// Please check its source code and implement error recovery when developing
// production applications.
sdmmc_card_t* card;
esp_err_t ret = esp_vfs_fat_sdmmc_mount("/sdcard", &host, &slot_config, &mount_config, &card);

if (ret != ESP_OK) {
    if (ret == ESP_FAIL) {
        ESP_LOGE(TAG, "Failed to mount filesystem. "
            "If you want the card to be formatted, set format_if_mount_failed = true.");
    } else {
        ESP_LOGE(TAG, "Failed to initialize the card (%s). "
            "Make sure SD card lines have pull-up resistors in place.", esp_err_to_name(ret));
    }
    return;
}
ESP_LOGE(TAG, "ZzZzZzZz      ********************************  ZZzZzZzZzZ***************************");
// Card has been initialized, print its properties
sdmmc_card_print_info(stdout, card);
ESP_LOGE(TAG, "ZzZzZzZz      ********************************  ZZzZzZzZzZ***************************");

/

Globals.h:

include

include

include <sys/unistd.h>

include <sys/stat.h>

include "esp_err.h"

include "esp_log.h"

include "esp_vfs_fat.h"

include "driver/sdmmc_host.h"

include "driver/sdspi_host.h"

include "sdmmc_cmd.h"

wifiscan.cpp:

FILE* f = fopen("/sdcard/Log_file.txt", "a+"); fprintf(f,"%02X:%02X:%02X:%02X:%02X:%02X | %u | %u | %02d\n", hdr->addr2[0],hdr->addr2[1],hdr->addr2[2],hdr->addr2[3],hdr->addr2[4],hdr->addr2[5],
(ppkt->rx_ctrl.timestamp)/1000000, sequencenumber1, ppkt->rx_ctrl.rssi); fprintf(f, "\n"); fclose(f);

AugustQu commented 4 years ago

Hi,

I'm back from my vacation and started working on the code again. Are you still interested in my approach to the SDcard-handling?

AugustQu commented 4 years ago

Hi,

this is what I did:

I renamed my existing project so that I can start with a clean directory

Then I added the code from here.

Then I added the code from here to the lib-directory and removed the "-master" from the name of the directorsy.

In src/hal I added a new file: ttgov21SD.h

In include I adeded a file sdcard.h and modified senddata.h

In src I:

For making the code compile with the enhancements:

this is the output of my test:

date, time, wifi, bluet 00.00.1970,00:01:09,2,0 00.00.1970,00:02:09,1,0 00.00.1970,00:03:09,2,0

Next step for me would be:

You may then check it. And naturally we can discuss the contents of the output or the code or

cyberman54 commented 4 years ago

yes, do add a branch for your modifications. I will review it.

Please do not create special hal files, but an option which can be used in all hal files. Like #define Display etc.

AugustQu commented 4 years ago

Hi,

I was not able to create a branch here so I copied the whole code from here to my repository, created a branch sdcard there and added my modifications. You will find my code here: https://github.com/AugustQu/PaxCounter

AugustQu commented 4 years ago

some questions:

1: can you legally include the lib for handling the sd-card in your code? Or do you have to add a link to the code plus a hint in the README-file?

2: in senddata.cpp I added some lines of code in the function SendPayload(). Maybe it should be put into the function sendData((). Don't know.

AugustQu commented 4 years ago

forgot: there are some restrictions on the cheap sd-card-readers:

these restrictions are valid as far as I know. Didn't check these, for me these restrictions are acceptable.

My code will create a file with the name PAXCOUNT.xx where xx goes from 00 to 99. It will start with 00, check to see if such a file already exists and if yes it will continue with the next number (up to 99 - in this case it will return no sd-card).

The data is written to the card and after 3 write-operations the data is flushed to the disk. So maybe the last 3 minutes of data get lost.

AugustQu commented 4 years ago

just created a pull-request from sdcard (on my repository) against development-branch here. Hope that's OK.

cyberman54 commented 4 years ago

I did a quick overlook.

cyberman54 commented 4 years ago

@AugustQu i merged your code contributions by hand. You find it here. Please use this branch for further developments. Thanks.

AugustQu commented 4 years ago

Thanks for your work.

I think the idea with the file ttgov21SD.h (in src/hal) is a bad one. I will remove this. A better approach would be a description of what to add to an existing hal-file in case one wants to use this feature.

One has to add these lines to the hal-file used:

define HAS_SDCARD 1 // this board has an SD-card-reader/writer

// Pins for SD-card

define SDCARD_CS (13) // fill in the correct numbers for your board

define SDCARD_MOSI (15)

define SDCARD_MISO (2)

define SDCARD_SCLK (14)

I will start writing an explanation. Where to put it? In README?

AugustQu commented 4 years ago

Hi,

blocking functions: I did not encounter any problems with my approach.

And 2nd: I'm totally new in programming for the ESP32-environment. So I will have to look what to do here.

cyberman54 commented 4 years ago

Usage of blocking functions may affect time critical routines. In paxcounter such are found in the lmic (lora) task, i.e. while waiting for RXing data. Your approach does not consider this. But it's mitigated since you used sendPayload() for writing to SD-card. SendPayload() is called by the interrupthandler, which considers if the device is in a time critical state. So far, so good.

But we may have to expect performance drop in MAC sniffing due to packet loss while the device is busy with writing to SD-card. Since this is done cyclic (controled by sendcycle) i think it's okay. So we keep this approach initially. A more elaborated approach would use queues.

cyberman54 commented 4 years ago

I will start writing an explanation. Where to put it? In README?

Please put it in generic.h and readme.md. You can also append credits at the end of readme.md ;-)

AugustQu commented 4 years ago

done.

I've added some lines to the README. I hope it's complete.

AugustQu commented 4 years ago

ah, my changes did not appear because:

You’re editing a file in a project you don’t have write access to. Submitting a change to this file will write it to a new branch in your fork AugustQu/ESP32-Paxcounter, so you can send a pull request.

AugustQu commented 4 years ago

OK, did the modifications again. You can find them in AugustQu/ESP32-Paxcounter/patch-2

I've also created a pull request against the development-branch here.

cyberman54 commented 4 years ago

I added sdcard support to branch development, and deleted branch sdcard. After i have tested sdcard support on different boards, i will merge #511 4e1472c with master and close this feature request.