Closed RickCavallaro closed 10 years ago
Sorry to hear you're running into problems! Here's what I had to do to get the example working. Hope this sheds some light on where you might be running into a problem:
The example works fine, repeatably, for me.
Here's a bunch of questions that will help us figuring out what may be going wrong:
Thanks very much Nathan. I'll give this a try and let you know what I find. I'm running mine on an Arduino Nano V3, but I can also try it on an Uno. I'm almost certain I'm already running at 9600 baud.
Meanwhile, let me take a crack at your questions...
What device are you attaching OpenLog to?
An Arduino Nano V3. But I will try an Uno as well if need be
What baud rate are you attempting to talk at?
9600
What voltage is everything running at?
5V. But I also tried using a Sparkfun Logic Level Converter as I assumed that could be the problem - rather than the reading itself. I later realized that couldn't be the problem because I get perfect results when asking for things like OpenLog status. The problem only occurs when I'm actually trying to read data from the SD card.
Have you tried system resetting the OpenLog?
No. But I have tried more than one OpenLog board - and your level-1 tech support folks got the same faulty results as me when they ran it on their hardware. I will try a reset.
Have you reconfigured OpenLog in any way (changed the command character)?
No.
Thanks again. I'll give this another try, making sure to follow your tips, and let you know how it goes.
Best regards.
Rick Cavallaro
From: Nathan Seidle notifications@github.com To: sparkfun/OpenLog OpenLog@noreply.github.com Cc: RickCavallaro dnrsw@yahoo.com Sent: Monday, August 4, 2014 1:58 PM Subject: Re: [OpenLog] Having trouble with the sample code... (#170)
Sorry to hear you're running into problems! Here's what I had to do to get the example working. Hope this sheds some light on where you might be running into a problem:
- Load example sketch onto RedBoard/Uno
- Pull SD card from openlog, open config on my computer and adjust the baud rate down to 9600bps. It was at 57600 for another one of my projects and the example sketch runs at 9600.
- Hook up 5 wires as defined at the top of the example sketch.
- Open serial com under Arduino IDE to view output. The example works fine, repeatably, for me. Here's a bunch of questions that will help us figuring out what may be going wrong:
- What device are you attaching OpenLog to?
- What baud rate are you attempting to talk at?
- What voltage is everything running at?
- Have you tried system resetting the OpenLog?
- Have you reconfigured OpenLog in any way (changed the command character)? — Reply to this email directly or view it on GitHub.
Thanks for continuing to figure this out! Only when reading data.... How odd. I'll see if I can learn anything from the setup that tech support was running.
There is nothing outlandish about your setup. I've never used a nano but should be fine. You don't need a level converter. The OpenLog is quite happy at 3.3V or 5V.
When you say:
It can write the file fine, but often fails to open the file to read - or succeeds, but seems to get a fair number of errors when it goes to read it.
What do you mean by it? The Nano or OpenLog. What I'm getting at is OpenLog might be working just fine and the Nano might be having parsing issues.
Nathan,
Thanks so much for helping with this. I watched your Google talk just the other day and loved it. I love what you guys are doing at Sparkfun.
I followed your directions carefully and it did in fact work just fine. But I remembered that it seemed to work OK for me when the file was quite small and/or had short lines. So I made three changes to the sketch:
1) I added two or three serial.println() calls just to show me how far it was getting. For some reason it initially was failing to get itself into read mode. 2) I hard-coded fileNumber so that it would write "LOG101.TXT" every time. Since the sketch opens the file to append, the file gets longer each time the sketch runs. 3) I added a line of text after the "Hi there! How are you today?" line. This is the line I added: "this is my addition for debugging. Let's just make one file rather than a different random file each time we run."
I've attached a zip file that includes the modified sketch and shows the results. When using the Arduino Nano to read from OpenLog and echo the data to the serial monitor you can see that it gets quite garbled. But when I read the SD card on my laptop you can see that it's actually logging perfectly every time.
What version of OpenLog firmware are you running?
I sent the "?" to OpenLog at the prompt and sent the resulting text to the serial monitor. It appears to be firmware version 3.11. As you can see, this also got pretty garbled.
OpenLog v3.11
Basic commands:
new
When you say: * It can write the file fine, but often fails to open the file to read - or succeeds, but seems to get a fair number of errors when it goes to read it. What do you mean by it? The Nano or OpenLog. What I'm getting at is OpenLog might be working just fine and the Nano might be having parsing issues. <<
I'm not sure how to answer that. I only communicate with OpenLog via the Arduino Nano. So I'm not able to tell whether the Arduino or OpenLog is at fault. I only know that I when I remove the SD card and read it on my laptop I always find the expected text without errors.
When you power up OpenLog and throw data at it can you read the text file back without errors? My guess is yes.
I have in the past thrown data at OpenLog for normal logging. When I put the card in my laptop the data appears to be fine. The only way I've tried to read the text back with OpenLog is via the sketch we're now discussing.
When you run the example script, what errors are you seeing?
I'm not getting any compile or run-time errors. The nature of the errors is simply that the text read back from the card (as displayed on the serial monitor through the Arduino dev environment) is garbled. But the data on the card itself is not garbled.
Reading back a file is fine but reading and trying to print the file at the same time is fraught with peril: The nano is spending too much time doing Serial.print() and not enough time listening to incoming characters from the OpenLog.
That may in fact be the problem. But that is exactly what I'm hoping to do. My hope is to read back the logged data on the card and send it to my laptop via bluetooth (since the whole thing will be in a waterproof enclosure). I presume I will run into the same problem there - yes?
Issue 169 may help.
Thanks. I just checked it out. It contains good information but doesn't seem to shed any light on the lost characters. I suspect you've hit on the problem with Serial.print() taking too much time. But that seems like one of the few things you can do with the data when reading back from the card. Is there a work-around? Maybe I could read a line from the card, print that to Serial, read the next line, etc.?
Thanks again for your help.
Best regards.
Rick Cavallaro 650-280-6898
From: Nathan Seidle notifications@github.com To: sparkfun/OpenLog OpenLog@noreply.github.com Cc: RickCavallaro dnrsw@yahoo.com Sent: Tuesday, August 5, 2014 9:37 AM Subject: Re: [OpenLog] Having trouble with the sample code... (#170)
Thanks for continuing to figure this out! Only when reading data.... How odd. I'll see if I can learn anything from the setup that tech support was running. There is nothing outlandish about your setup. I've never used a nano but should be fine. You don't need a level converter. The OpenLog is quite happy at 3.3V or 5V.
- What version of OpenLog firmware are you running? Type ? at the prompt and it will report at the top. If you're above about v3.1 then you're doing ok. I just need to be sure what monster we're up against.
- When you say: It can write the file fine, but often fails to open the file to read - or succeeds, but seems to get a fair number of errors when it goes to read it. What do you mean by it? The Nano or OpenLog. What I'm getting at is OpenLog might be working just fine and the Nano might be having parsing issues.
- When you power up OpenLog and throw data at it can you read the text file back without errors? My guess is yes.
- When you run the example script, what errors are you seeing? The example is there just to show how to pass commands back and forth. Reading back a file is fine but reading and trying to print the file at the same time is fraught with peril: The nano is spending too much time doing Serial.print() and not enough time listening to incoming characters from the OpenLog. Issue 169 may help. — Reply to this email directly or view it on GitHub.
Hi Rick,
Your example output is really garbled in some funky ways - makes me believe you may have some array indexing errors in your Nano code. I didn't see a zip file? Maybe try again. I'd like to have a look at your code.
Excellent feedback, I think I've got a good grasp of what's going on.
I highly recommend an FTDI Basic and a crossover board. This will allow you to connect directly to the OpenLog so that you can configure it, throw lots of data at it, read lots of data back and see what exactly the OpenLog is doing.
The fact that OpenLog is recording everything correctly means the OpenLog is performing as expected. I'm really starting to believe the error is stemming from the Nano.
Here's the problem: Reading serial from the OpenLog via software serial takes a large amount of processor overhead. Let's say the Nano is spending 98% of its time monitoring the incoming serial characters from the OpenLog. The Nano then has to spend time sending out those characters over serial. Every time the Nano reports something that is time not spent listening to the OpenLog and that's when characters get missed.
For example, when the Nano is printing this
OpenLog v3.11
Basic commands:
new <file>: Creates <file>
The Nano is not monitoring the incoming serial from the OpenLog so when the Nano gets done printing stuff it goes back to listening and misses a few dozen incoming characters from the OpenLog. The solution is to print less and listen more. The example code mentions this a few times.
And as a final question (sorry!) - what is your overall application? Most people use OpenLog for logging. Reading back from OpenLog is do-able but requires more processor power.
Thanks again for the response.
makes me believe you may have some array indexing errors in your Nano code.
I wondered that as well - but I really haven't changed any of that from the example code you provided. I just now repeated the problem using your example code with only a single change. I added "fileNumber = 110;". This way it always writes to and appends to "log110.txt" rather than creating a random file. Another really inexplicable thing is that your example code doesn't work for me until I add that line. It writes the file, but it gets stuck trying to open the file to read.
Without my change I get this on the serial monitor in the Arduino dev env.: OpenLog online
Random file created: log105.txt Text written to file Reading file contents: And it just hangs there. When I add "fileNumber = 110;" it does succeed in opening the file to read - and reads it successfully. But when I run the program repeatedly it continues to append to the file, and it starts giving garbled output:
OpenLog online
Random file created: log110.txt Text written to file Reading file contents:
Hi there! How are you today? Voltage: 3.50 Voltage: 4.50 ...Hi there! How are you today? Voltage: 3.50 Voltage: 4.50 ...Hi there! How are you today? Voltagutoa? Vlag: .50 oltag: 450 ..Hi tere! ow ar you oday? Voltae: 3.0 Votage:4.50 ...Hithere How re g:350 olae 450 ...
Card type: SD2 Manufacturer ID: 12 OEM ID: 4V Product: SD
Version: 2.0 Serial number: 1677197312 Manufacturing date: 3/2012 Card Size: 976896 KB
File read complete Yay! Reading the file on my laptop shows that it is not garbled at all. Interestingly the response to "readDisk()" is also never garbled - only the data it's actually reading from the card gets garbled.
I'd like to have a look at your code.
Here's the code I just ran. It's your example with the one line change I mentioned:
/*
Example of reading the disk properties on OpenLog By: Nathan Seidle SparkFun Electronics Date: September 22nd, 2013 License: This code is public domain but you buy me a beer if you use this and we meet someday (Beerware license).
This is an example of issuing the 'disk' command and seeing how big the current SD card is. Connect the following OpenLog to Arduino: RXI of OpenLog to pin 2 on the Arduino TXO to 3 GRN to 4 VCC to 5V GND to GND This example code assumes the OpenLog is set to operate at 9600bps in NewLog mode, meaning OpenLog should power up and output '12<'. This code then sends the three escape characters and then sends the commands to create a new random file called log###.txt where ### is a random number from 0 to 999. The example code will then read back the random file and print it to the serial terminal. This code assume OpenLog is in the default state of 9600bps with ASCII-26 as the esacape character. If you're unsure, make sure the config.txt file contains the following: 9600,26,3,0 Be careful when sending commands to OpenLog. println() sends extra newline characters that cause problems with the command parser. The new v2.51 ignores \n commands so it should be easier to talk to on the command prompt level. This example code works with all OpenLog v2 and higher. */
include
//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- //Connect TXO of OpenLog to pin 3, RXI to pin 2 SoftwareSerial OpenLog(3, 2); //Soft RX on 3, Soft TX out on 2 //SoftwareSerial(rxPin, txPin)
int resetOpenLog = 4; //This pin resets OpenLog. Connect pin 4 to pin GRN on OpenLog. //-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
int statLED = 13;
float dummyVoltage = 3.50; //This just shows to to write variables to OpenLog
void setup() { pinMode(statLED, OUTPUT); Serial.begin(9600);
setupOpenLog(); //Resets logger and waits for the '<' I'm alive character Serial.println("OpenLog online"); }
void loop() {
randomSeed(analogRead(A0)); //Use the analog pins for a good seed value int fileNumber = random(999); //Select a random file #, 0 to 999 fileNumber = 110; // my change char fileName[12]; //Max file name length is "12345678.123" (12 characters) sprintf(fileName, "log%03d.txt", fileNumber);
gotoCommandMode(); //Puts OpenLog in command mode createFile(fileName); //Creates a new file called log###.txt where ### is random
Serial.print("Random file created: "); Serial.println(fileName);
//Write something to OpenLog OpenLog.println("Hi there! How are you today?"); OpenLog.print("Voltage: "); OpenLog.println(dummyVoltage); dummyVoltage++; OpenLog.print("Voltage: "); OpenLog.println(dummyVoltage);
Serial.println("Text written to file"); Serial.println("Reading file contents:"); Serial.println();
//Now let's read back gotoCommandMode(); //Puts OpenLog in command mode readFile(fileName); //This dumps the contents of a given file to the serial terminal
//Now let's read back readDisk(); //Check the size and stats of the SD card
Serial.println(); Serial.println("File read complete");
//Infinite loop Serial.println("Yay!"); while(1) { digitalWrite(13, HIGH); delay(250); digitalWrite(13, LOW); delay(250); } }
//Setups up the software serial, resets OpenLog so we know what state it's in, and waits //for OpenLog to come online and report '<' that it is ready to receive characters to record void setupOpenLog(void) { pinMode(resetOpenLog, OUTPUT); OpenLog.begin(9600);
//Reset OpenLog digitalWrite(resetOpenLog, LOW); delay(100); digitalWrite(resetOpenLog, HIGH);
//Wait for OpenLog to respond with '<' to indicate it is alive and recording to a file while(1) { if(OpenLog.available()) if(OpenLog.read() == '<') break; } }
//This function creates a given file and then opens it in append mode (ready to record characters to the file) //Then returns to listening mode void createFile(char *fileName) {
//Old way OpenLog.print("new "); OpenLog.print(fileName); OpenLog.write(13); //This is \r
//New way //OpenLog.print("new "); //OpenLog.println(filename); //regular println works with OpenLog v2.51 and above
//Wait for OpenLog to return to waiting for a command while(1) { if(OpenLog.available()) if(OpenLog.read() == '>') break; }
OpenLog.print("append "); OpenLog.print(fileName); OpenLog.write(13); //This is \r
//Wait for OpenLog to indicate file is open and ready for writing while(1) { if(OpenLog.available()) if(OpenLog.read() == '<') break; }
//OpenLog is now waiting for characters and will record them to the new file }
//Reads the contents of a given file and dumps it to the serial terminal //This function assumes the OpenLog is in command mode void readFile(char *fileName) {
//Old way OpenLog.print("read "); OpenLog.print(fileName); OpenLog.write(13); //This is \r
//New way //OpenLog.print("read "); //OpenLog.println(filename); //regular println works with OpenLog v2.51 and above
//The OpenLog echos the commands we send it by default so we have 'read log823.txt\r' sitting //in the RX buffer. Let's try to not print this. while(1) { if(OpenLog.available()) if(OpenLog.read() == '\r') break; }
//This will listen for characters coming from OpenLog and print them to the terminal //This relies heavily on the SoftSerial buffer not overrunning. This will probably not work //above 38400bps. //This loop will stop listening after 1 second of no characters received for(int timeOut = 0 ; timeOut < 1000 ; timeOut++) { while(OpenLog.available()) { char tempString[100]; int spot = 0; while(OpenLog.available()) { tempString[spot++] = OpenLog.read(); if(spot > 98) break; } tempString[spot] = '\0'; Serial.write(tempString); //Take the string from OpenLog and push it to the Arduino terminal timeOut = 0; }
delay(1); }
//This is not perfect. The above loop will print the '.'s from the log file. These are the two escape characters //recorded before the third escape character is seen. //It will also print the '>' character. This is the OpenLog telling us it is done reading the file.
//This function leaves OpenLog in command mode }
//Check the stats of the SD card via 'disk' command //This function assumes the OpenLog is in command mode void readDisk() {
//Old way OpenLog.print("disk"); OpenLog.write(13); //This is \r
//New way //OpenLog.print("read "); //OpenLog.println(filename); //regular println works with OpenLog v2.51 and above
//The OpenLog echos the commands we send it by default so we have 'disk\r' sitting //in the RX buffer. Let's try to not print this. while(1) { if(OpenLog.available()) if(OpenLog.read() == '\r') break; }
//This will listen for characters coming from OpenLog and print them to the terminal //This relies heavily on the SoftSerial buffer not overrunning. This will probably not work //above 38400bps. //This loop will stop listening after 1 second of no characters received for(int timeOut = 0 ; timeOut < 1000 ; timeOut++) { while(OpenLog.available()) { char tempString[100]; int spot = 0; while(OpenLog.available()) { tempString[spot++] = OpenLog.read(); if(spot > 98) break; } tempString[spot] = '\0'; Serial.write(tempString); //Take the string from OpenLog and push it to the Arduino terminal timeOut = 0; }
delay(1); }
//This is not perfect. The above loop will print the '.'s from the log file. These are the two escape characters //recorded before the third escape character is seen. //It will also print the '>' character. This is the OpenLog telling us it is done reading the file.
//This function leaves OpenLog in command mode }
//This function pushes OpenLog into command mode void gotoCommandMode(void) { //Send three control z to enter OpenLog command mode //Works with Arduino v1.0 OpenLog.write(26); OpenLog.write(26); OpenLog.write(26);
//Wait for OpenLog to respond with '>' to indicate we are in command mode while(1) { if(OpenLog.available()) if(OpenLog.read() == '>') break; } }
I highly recommend an FTDI Basic and a crossover board.
I like that idea. I will probably try that. I think the other thing I'll try is reading one byte at a time from OpenLog using "readBytes()" so that it's not trying to read and print at the same time. I presume that "serial.read()" is telling openLog to stream data - and the timing errors you mentioned have the Arduino failing to keep up. Is this right?
Here's the problem: Reading serial from the OpenLog via software serial takes a large amount of processor overhead. Let's say the Nano is spending 98% of its time monitoring the incoming serial characters from the OpenLog. The Nano then has to spend time sending out those characters over serial. Every time the Nano reports something that is time not spent listening to the OpenLog and that's when characters get missed.
<<
I suspect you're right. I just now changed the code further so that it no longer appends to the file and it no longer prints the file contents to the Arduino dev. env. serial monitor. It simply counts the bytes in the file and reports the final number to the serial monitor. It consistently reports reading 428 bytes. This suggests to me that it's not making errors. When I read the file on my laptop it shows as 504 bytes, but I'm guessing that's somehow counting the difference between a
what is your overall application? Most people use OpenLog for logging. Reading back from OpenLog is do-able but requires more processor power.
I'm making a jump height meter for kitesurfing. It will use the Arduino with the MS5611 to read altitude. It will log jumps to openLog, and it will then stream the data to the laptop via bluetooth. This lets me put the whole thing in a completely water-tight bladder with no wires or anything hanging out.
I have used it in the past in exactly the way you describe. It worked great. We developed a wind powered vehicle that goes faster than the wind both directly upwind and directly downwind. We had to log a lot of wind and GPS data. But we read the data on a laptop. Here's one of our downwind runs...
https://www.youtube.com/watch?v=5CcgmpBGSCI
Thanks again for all your help. I think I'm also going to try an SPI flash IC for this application.
Best regards.
Rick Cavallaro
That is an awesome video! Thanks for sharing. SO COOL. Salt flats in Utah? Looks like fun.
The application makes a lot more sense to me now (record, read, report, need to be all sealed up). I'll try to write a tighter read example that drops less characters. I think it's do-able, I just never had my head wrapped around an application.
I just tried reading it byte-by-byte. I turned echo off and I actually read 6 bytes for each request (I guess the response includes a prompt,
Here's my current code for reading the file byte-by-byte:
void readFile(char *fileName)
{ int i; int which_char; int byte_count = 0; char tempChar; int start = 0; int found_char; while(1) { // Tell OpenLog to send one specific byte OpenLog.print("read "); OpenLog.print(fileName); OpenLog.print(" "); OpenLog.print(start); OpenLog.print(" 1"); OpenLog.write(13); //This is \r
which_char = 0; // I only want one character, but it seems to print out six characters (a prompt,
, etc. and I only want character #3), so read characters until I have all 6 or no more are waiting. while(1) { // Now read that byte found_char = 0; for(i=0; i<1000; i++) { if (OpenLog.available()) { tempChar = OpenLog.read(); which_char++; found_char = 1; break; } } if (found_char == 0 || which_char == 6) break; // no more characters for this request if (which_char == 3) Serial.write(tempChar); } start += 1; } }
Sorry to have put you to so much trouble for my silly problem. But I really appreciate it. I think the SPI flash IC is probably the better approach for this application since there's no need for physically removing the memory/card. I think I'll give that a try.
Thanks again.
Rick Cavallaro
From: Nathan Seidle notifications@github.com To: sparkfun/OpenLog OpenLog@noreply.github.com Cc: RickCavallaro dnrsw@yahoo.com Sent: Sunday, August 10, 2014 7:23 PM Subject: Re: [OpenLog] Having trouble with the sample code... (#170)
That is an awesome video! Thanks for sharing. SO COOL. Salt flats in Utah? Looks like fun. The application makes a lot more sense to me now (record, read, report, need to be all sealed up). I'll try to write a tighter read example that drops less characters. I think it's do-able, I just never had my head wrapped around an application. — Reply to this email directly or view it on GitHub.
Hi Rick - no problem at all. It gives me a chance to write some code :)
I committed a new large file example that successfully reads a large file from OpenLog at 9600, connected to the hardware RX/TX on the Arduino. It then spits those characters back out over software serial at 14400 or higher. If you can setup your Bluetooth connection to work at a higher baud rate (say 57600) then this example should show you the basics.
Going to close the issue. Please let me know if you run into more problems - and send us more project updates!
Thanks very much. I'll check it out. If it's not already - you should put a link to it on the OpenLog page on your website.
Rick C.
From: Nathan Seidle notifications@github.com To: sparkfun/OpenLog OpenLog@noreply.github.com Cc: RickCavallaro dnrsw@yahoo.com Sent: Monday, August 11, 2014 3:19 PM Subject: Re: [OpenLog] Having trouble with the sample code... (#170)
Hi Rick - no problem at all. It gives me a chance to write some code :) I committed a new large file example that successfully reads a large file from OpenLog at 9600, connected to the hardware RX/TX on the Arduino. It then spits those characters back out over software serial at 14400 or higher. If you can setup your Bluetooth connection to work at a higher baud rate (say 57600) then this example should show you the basics. Going to close the issue. Please let me know if you run into more problems - and send us more project updates! — Reply to this email directly or view it on GitHub.
I'm using the sample code provided here... https://github.com/sparkfun/OpenLog/blob/master/firmware/examples/OpenLog_ReadExample/OpenLog_ReadExample.ino
It's supposed to:
It can write the file fine, but often fails to open the file to read - or succeeds, but seems to get a fair number of errors when it goes to read it.
On the other hand if I write a new file, and then open one of the older files to read, it seems to handle both fine - and without errors.
Any tips would be greatly appreciated. Thanks.