problem in getting csv file

[Tremito96]

I manged to get the CSI data on the monitor. But i had to use idf.py instead of "make"(examaple-> idf.py menuconfig).. however that did not work for generating the csv file. can anyone help? this was the error given. " 'grep' is not recognized as an internal or external command,operable program or batch file." im pretty new to ESP IDF..

[StevenMHernandez]

grep is command found in UNIX systems (Linux and MacOS for example) but does not seem to be available on Windows by default. Have you searched for solutions to this error online? For example:

• https://superuser.com/questions/1371375/grep-command-fails-on-one-windows-pc-but-works-on-other-windows-pc-w-same-conf
• https://superuser.com/a/300825

I don't have exact instructions with windows (I don't have access to a windows computer to test), but one of those links should help.

[ENB213-xiaoh]

Hi Steven I encountered a new problem. I can get CSI by running make monitor, all CSI will be showing in the terminal. But, I couldn't get any CSI by running make monitor | grep "CSI_DATA" > test.csv. Please kindly advise. Thank you

[StevenMHernandez]

Does it create a test.csv file? Or is the file empty? How long do you run the code? Notice, when you run that command it ONLY writes to a file, but does not show up on the terminal

[ENB213-xiaoh]

Thank you for your reply. I have solved it. No idea why I have to use that python code located in utils to save it into files. But, it works.

[coco40811]

Excuse me,I also have the problem in getting csv file. When I run the command(make monitor | grep "CSIDATA" > my-experiment-file.csv),it return the error "stdout is not a tty".I don't know how to solve it.Could anyone help?

[ENB213-xiaoh]

Use the second one with python code in utils will be okay

[coco40811]

Thanks for reply,but I still have the problem when I using the second one.Here is the text I got. "stdout is not a tty" "make: *** [/home/USER/esp/esp-idf/components/esptool_py/Makefile.projbuild:114: monitor] Error 1"

[StevenMHernandez]

Hmm, this is weird. This happens when you call make monitor by itself right? This error seems to be coming from esp-idf, it is unrelated to the python code in this github repo.

The only reference I can find is (https://www.esp32.com/viewtopic.php?t=1462) which mentions (stdout is not a tty).

Are you sure that you installed esp-idf correctly? Were you able to run make flash?

Also, are you able to run make flash and/or make monitor on any of the examples in the esp-idf that you installed (https://github.com/espressif/esp-idf/tree/master/examples)?

[coco40811]

No,it happend only when I run "make monitor | grep "CSIDATA" > my-experiment-file.csv" or "make monitor | python ./_utils/serial_append_time.py > my-experiment-file.csv" .I am sure that "make monitor" work successfully.when I run "make monitor" ,I can get csi data on the board.So I'm still figuring it out what goes wrong.

[ENB213-xiaoh]

You may use the home directory to refer to that python code, for example, /home/yourname/..../_utils/.......

[StevenMHernandez]

Great, I am glad that it is working like that.

We need to break down the error into the smallest parts. This will help us identify what parts of the command are working and which ones are not working. We see that make monitor works so let's break it down and test the other parts.

It seems like the only other thing common to the two scripts that you ran is the > my-experiment-file.csv part. So can you try the following and confirm if they work or if they fail:

1. echo "testing" > new_file.csv
2. make monitor > new_file.csv
3. make monitor | grep "CSI_DATA"
4. make monitor | grep "CSI_DATA" > new_file.csv
5. make monitor | cat > new_file.csv

Can you tell me a bit about your development environment: what OS are you using, what terminal/shell.

Side note: when adding code to a comment, you can use the ` to format it so that it is easier to read :) (see: https://docs.github.com/en/github/writing-on-github/basic-writing-and-formatting-syntax#quoting-code)

[coco40811]

Following is my result and terminal/shell. My OS is win10.

[StevenMHernandez]

OK, so the minimum error appears whenever you try to pipe or redirect the output stream from make monitor. I really want to know what is going on, but I want to emphasis that this is not a problem with the ESP32-CSI-Tool. Instead, it looks like an issue with using pipes in mingw32 with esp-idf.

I would recommend asking other people using esp-idf on windows if they have seen this problem. I am sure someone else has used windows and has seen this problem. (I do not use windows)

If you do ask over there, please report back. The answer will help other people here too. Thanks.

[StevenMHernandez]

These links look promising:

• https://blog.stigok.com/2017/03/13/node-executable-on-windows-mingw-output-is-not-a-tty.html
• https://stackoverflow.com/questions/45112889/bash-node-js-stdin-stdout-redirection-error-not-a-tty

Do they help?

[ahmetanbar]

I

These links look promising:

• https://blog.stigok.com/2017/03/13/node-executable-on-windows-mingw-output-is-not-a-tty.html
• https://stackoverflow.com/questions/45112889/bash-node-js-stdin-stdout-redirection-error-not-a-tty

Do they help?

I try this, but they don't work. I see that latest version of idf has a monitoring system that has saving capabilitiy by using shortcuts. https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/tools/idf-monitor.html

Have you test this projest for latest version?

[StevenMHernandez]

Are you talking about ctrl+L? In my opinion requiring keyboard commands is not a great idea. I would prefer it to be more automated in some way.

Unfortunately, the issues here seem to be Windows related and I just do not have access to a Windows computer to actually try it out. I can't do it myself, I need help! :)

If anyone has got it working on Windows, please let me know in #18. Thanks!

[ahmetanbar]

I hope that I will figure it out... There are some situations that I don't understand. I hope I will solve it.

[NiyamPoudel]

My CSV data is empty. What could have gone wrong?

[StevenMHernandez]

@NiyamPoudel Did you try any of the suggestions in this thread?

[NiyamPoudel]

Turns out, it takes quite a while to get the data. I have been running it for 10 minutes now. So far I have only gotten 1 row of data.

nimai@nimai:~/esp/esp32-csi-tool/passive$ make flash monitor
Toolchain path: /home/nimai/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a
Compiler version: 5.2.0
Python requirements from /home/nimai/esp/esp-idf/requirements.txt are satisfied.

App "passive" version: f8f95a5
Flashing binaries to serial port /dev/ttyUSB0 (app at offset 0x10000)...
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting.......
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 7c:9e:bd:47:81:28
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0220
Compressed 25632 bytes to 15269...
Wrote 25632 bytes (15269 compressed) at 0x00001000 in 1.4 seconds (effective 151.6 kbit/s)...
Hash of data verified.
Compressed 685744 bytes to 416394...
Wrote 685744 bytes (416394 compressed) at 0x00010000 in 36.9 seconds (effective 148.8 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 1802.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
MONITOR
--- idf_monitor on /dev/ttyUSB0 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:6744
load:0x40078000,len:12072
load:0x40080400,len:6708
entry 0x40080778
I (71) boot: Chip Revision: 1
I (71) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (39) boot: ESP-IDF v3.3.1 2nd stage bootloader
I (39) boot: compile time 09:14:44
I (39) boot: Enabling RNG early entropy source...
I (43) boot: SPI Speed      : 40MHz
I (47) boot: SPI Mode       : DIO
I (51) boot: SPI Flash Size : 4MB
I (55) boot: Partition Table:
I (59) boot: ## Label            Usage          Type ST Offset   Length
I (66) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (74) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (81) boot:  2 factory          factory app      00 00 00010000 00100000
I (89) boot: End of partition table
I (93) boot_comm: chip revision: 1, min. application chip revision: 0
I (100) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x1e384 (123780) map
I (152) esp_image: segment 1: paddr=0x0002e3ac vaddr=0x3ffb0000 size=0x01c64 (  7268) load
I (156) esp_image: segment 2: paddr=0x00030018 vaddr=0x400d0018 size=0x7554c (480588) map
0x400d0018: _stext at ??:?

I (328) esp_image: segment 3: paddr=0x000a556c vaddr=0x3ffb1c64 size=0x013f0 (  5104) load
I (330) esp_image: segment 4: paddr=0x000a6964 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /home/nimai/esp/esp-idf/components/freertos/xtensa_vectors.S:1779

I (335) esp_image: segment 5: paddr=0x000a6d6c vaddr=0x40080400 size=0x10918 ( 67864) load
I (382) boot: Loaded app from partition at offset 0x10000
I (382) boot: Disabling RNG early entropy source...
I (382) cpu_start: Pro cpu up.
I (386) cpu_start: Application information:
I (391) cpu_start: Project name:     passive
I (396) cpu_start: App version:      f8f95a5
I (401) cpu_start: Compile time:     Jul 23 2021 09:14:47
I (407) cpu_start: ELF file SHA256:  854d7d494c9e56c3...
I (413) cpu_start: ESP-IDF:          v3.3.1
I (417) cpu_start: Starting app cpu, entry point is 0x40081198
0x40081198: call_start_cpu1 at /home/nimai/esp/esp-idf/components/esp32/cpu_start.c:269

I (0) cpu_start: App cpu up.
I (428) heap_init: Initializing. RAM available for dynamic allocation:
I (435) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (441) heap_init: At 3FFB9108 len 00026EF8 (155 KiB): DRAM
I (447) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (453) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (460) heap_init: At 40090D18 len 0000F2E8 (60 KiB): IRAM
I (466) cpu_start: Pro cpu start user code
I (149) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (215) gpio: GPIO[13]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0 
E (257) sdmmc_sd: sdmmc_init_sd_if_cond: send_if_cond (1) returned 0x108
I (257) gpio: GPIO[15]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
I (263) gpio: GPIO[2]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
I (272) gpio: GPIO[14]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
E (281) sd.h: Failed to initialize the card (ESP_ERR_INVALID_RESPONSE).   If you do not have an SD card attached, please ignore this message.  Make sure SD card lines have pull-up resistors in place.
I (303) wifi: wifi driver task: 3ffbf714, prio:23, stack:3584, core=0
I (306) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (315) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (350) wifi: wifi firmware version: ac331d7
I (350) wifi: config NVS flash: enabled
I (350) wifi: config nano formating: disabled
I (351) wifi: Init dynamic tx buffer num: 32
I (355) wifi: Init data frame dynamic rx buffer num: 32
I (360) wifi: Init management frame dynamic rx buffer num: 32
I (365) wifi: Init management short buffer num: 32
I (370) wifi: Init static rx buffer size: 2212
I (374) wifi: Init static rx buffer num: 10
I (378) wifi: Init dynamic rx buffer num: 32
I (477) phy: phy_version: 4102, 2fa7a43, Jul 15 2019, 13:06:06, 0, 0
I (478) wifi: mode : null
I (479) wifi: ic_enable_sniffer
type,role,mac,rssi,rate,sig_mode,mcs,bandwidth,smoothing,not_sounding,aggregation,stbc,fec_coding,sgi,noise_floor,ampdu_cnt,channel,secondary_channel,local_timestamp,ant,sig_len,rx_state,real_time_set,real_timestamp,len,CSI_DATA
CSI_DATA,PASSIVE,34:68:95:4E:8C:B3,-47,11,0,0,0,0,0,0,0,0,0,-92,0,8,0,143087727,0,30,0,0,143.391637,128,[30 -32 1 0 18 9 21 8 18 8 20 8 21 7 22 5 19 5 21 5 18 4 20 5 20 4 21 3 21 4 21 2 22 2 24 -1 21 1 22 0 20 2 24 0 24 1 24 1 25 1 25 2 24 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 6 20 8 21 9 21 10 18 10 18 11 18 11 17 12 17 12 17 12 18 13 18 13 18 12 17 12 15 13 19 12 17 12 15 11 17 12 16 11 15 10 18 11 14 10 20 11 20 11 18 10 ]

[StevenMHernandez]

It sounds like you have no CSI communication going on in the environment. If you are not comfortable with this tool, I would recommend trying active_ap + active_sta before you try out the passive method. Otherwise, you will not be able to understand what is going on and you will not be able to solve any problems on your own.

[StevenMHernandez]

I think this issue breaks down to "Windows does not have grep". In which case, #18 should handle this.

There are other irrelevant conversations in here so I am going to close this issue. If you have more questions, please ask.

[olliec28]

I think this issue breaks down to "Windows does not have grep". In which case, #18 should handle this.

There are other irrelevant conversations in here so I am going to close this issue. If you have more questions, please ask.

I am using macos and I currently have one esp32 in active_ap directory and its not collecting any data I don't believe. The output log is:

Olivers-MacBook-Pro:active_ap olivercustance$ idf.py flash monitor Executing action: flash Serial port /dev/cu.usbserial-0001 Connecting................. Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting.... Detecting chip type... ESP32 Running ninja in directory /Users/olivercustance/esp32-csi-tool/active_ap/build Executing "ninja flash"... [1/4] Performing build step for 'bootloader' ninja: no work to do. [3/4] cd /Users/olivercustance/esp/esp...nents/esptool_py/run_serial_tool.cmake esptool.py esp32 -p /dev/cu.usbserial-0001 -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 2MB 0x8000 partition_table/partition-table.bin 0x1000 bootloader/bootloader.bin 0x10000 active-ap.bin esptool.py v3.3.2-dev Serial port /dev/cu.usbserial-0001 Connecting..... Chip is ESP32-D0WDQ6 (revision 1) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: 0c:b8:15:cd:af:50 Uploading stub... Running stub... Stub running... Changing baud rate to 460800 Changed. Configuring flash size... Flash will be erased from 0x00008000 to 0x00008fff... Flash will be erased from 0x00001000 to 0x00007fff... Flash will be erased from 0x00010000 to 0x000e7fff... Compressed 3072 bytes to 103... Writing at 0x00008000... (100 %) Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.1 seconds (effective 314.4 kbit/s)... Hash of data verified. Compressed 26224 bytes to 16165... Writing at 0x00001000... (100 %) Wrote 26224 bytes (16165 compressed) at 0x00001000 in 0.9 seconds (effective 234.4 kbit/s)... Hash of data verified. Compressed 882400 bytes to 515905... Writing at 0x00010000... (3 %) Writing at 0x0001c8da... (6 %) Writing at 0x00025e83... (9 %) Writing at 0x00039410... (12 %) Writing at 0x0003f6ef... (15 %) Writing at 0x00045451... (18 %) Writing at 0x0004abbb... (21 %) Writing at 0x00050446... (25 %) Writing at 0x00055a6a... (28 %) Writing at 0x0005b18f... (31 %) Writing at 0x00060b01... (34 %) Writing at 0x000665c5... (37 %) Writing at 0x0006b88d... (40 %) Writing at 0x00070b68... (43 %) Writing at 0x00075e9d... (46 %) Writing at 0x0007b420... (50 %) Writing at 0x000809fb... (53 %) Writing at 0x00086319... (56 %) Writing at 0x0008c0e3... (59 %) Writing at 0x00091a02... (62 %) Writing at 0x000974d9... (65 %) Writing at 0x0009dbcc... (68 %) Writing at 0x000a4f39... (71 %) Writing at 0x000ad9f2... (75 %) Writing at 0x000b4dd4... (78 %) Writing at 0x000baf63... (81 %) Writing at 0x000c3ef1... (84 %) Writing at 0x000ca2fc... (87 %) Writing at 0x000d2e32... (90 %) Writing at 0x000d9ce2... (93 %) Writing at 0x000df81f... (96 %) Writing at 0x000e4a90... (100 %) Wrote 882400 bytes (515905 compressed) at 0x00010000 in 13.2 seconds (effective 534.1 kbit/s)... Hash of data verified.

Leaving... Hard resetting via RTS pin... Executing action: monitor Serial port /dev/cu.usbserial-0001 Connecting.... Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting.... Detecting chip type... ESP32 Running idf_monitor in directory /Users/olivercustance/esp32-csi-tool/active_ap Executing "/Users/olivercustance/.espressif/python_env/idf4.3_py3.8_env/bin/python /Users/olivercustance/esp/esp-idf/tools/idf_monitor.py -p /dev/cu.usbserial-0001 -b 1552000 --toolchain-prefix xtensa-esp32-elf- /Users/olivercustance/esp32-csi-tool/active_ap/build/active-ap.elf -m '/Users/olivercustance/.espressif/python_env/idf4.3_py3.8_env/bin/python' '/Users/olivercustance/esp/esp-idf/tools/idf.py'"... /Users/olivercustance/esp/esp-idf/tools/idf_monitor.py:518: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if StrictVersion(serial.VERSION) < StrictVersion('3.3.0'): --- idf_monitor on /dev/cu.usbserial-0001 1552000 --- --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

Thanks, Ollie

[StevenMHernandez]

EDIT: Discussion moved to #75

@olliec28, you are not using grep, so I am not sure what you are referencing in the quote. Also, I do not think the command you are using is trying to save to a csv file, so I think this is the wrong issue for your issue. Unless I am mistaken.

Either way, it looks like NOTHING is being printed by the ESP32 itself (there should be generic printed sentences after:

--- idf_monitor on /dev/cu.usbserial-0001 1552000 --- --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

As such, my best guess would be that you have not set the baud rate correctly in your settings. From what I can tell, this is not a problem with the ESP32-CSI-Tool, it is just a configuration problem. Once you can figure out a way to make the ESP32 print anything, you should be good-to-go.

[olliec28]

EDIT: Discussion moved to #75

@olliec28, you are not using grep, so I am not sure what you are referencing in the quote. Also, I do not think the command you are using is trying to save to a csv file, so I think this is the wrong issue for your issue. Unless I am mistaken.

Either way, it looks like NOTHING is being printed by the ESP32 itself (there should be generic printed sentences after:

--- idf_monitor on /dev/cu.usbserial-0001 1552000 --- --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

As such, my best guess would be that you have not set the baud rate correctly in your settings. From what I can tell, this is not a problem with the ESP32-CSI-Tool, it is just a configuration problem. Once you can figure out a way to make the ESP32 print anything, you should be good-to-go.

Im delighted to say it works perfectly fine now. I split the flash and monitor step into two commands. For Mac there's a few extra steps which is required to get csi data:

1. idf.py -p PORT NAME -b BAUD RATE flash
2. idf.py -p PORT NAME monitor
3. idf.py -p PORT NAME monitor | grep "CSI_DATA" > my-experiment-file.csv

Hope this helps any Mac users to avoid getting empty csv files