Improvement of stdio.h functions

[shiftleftplusone]

hi, I would appreciate if the number of stdio.h functions for file access could be improved. For file access we basically just have (file).print, println, .write, and just .read (by single bytes). But fprintf and fscanf would be appriciated, too.

E.g.,that's why I just faked fgets() to something which mimics the ANSI function. It's been programmed quick and dirty, but IMO it's already an improvement. You may polish it up for your libs and add it for use by SD if you wish.

char * fgets ( char * str, int32_t num, File * stream ) {
  int32_t i;
  i=0;
  strcpy(str, "");
  while ( stream->available() && (i < num)   ) {  
     str[i]=stream->read() ;
     if(str[i] >=  ' ')  ++i;
     else
     if(str[i] == '\0') goto ok; // end of string: \0
     else
     if(str[i] == '\n')   {      // end of string: CR
         //str[i] = '\0';          // optional: overwrite CR by '\0'
         str[++i] = '\0';      // insert additional 0-termination after CR (== ANSI C)
         goto ok;
     }  
  } 
  ok: return(str);
}

function call: 
char sdata[30];
File myFile;
fgets ( sdata, 20, &myFile ); 
or
char * sptr;
sptr = fgets ( sdata, 20, &myFile ); 

I just am trying to mimic more functions using multiple variable parameter lists ( ... ) provided by

#include <stdarg.h>
#include <stdio.h>

e.g.,

char * fprintf(File * stream,  char fmtstr[], ... )

but of course it would be better to access the "real" stdio.h functionality instead of a faked and limited homebrewed one.

ps, the source code gets always corrupted after uploading it here !! :(

[peabo7]

Your fgets doesn't do what real fgets does, so you should call it something else, if you use it at all.

There are several problems:

It has a buffer overrun because it doesn't pay attention to the num parameter. It never detects end of file, because 0 is not the indicator for end of file (-1 is, which is why you need to read into a int16_t). The handling of end of file returning NULL is quite specific, in the case where end of file happens before reading any characters. The loop terminates prematurely if stream->available() returns 0.

Try this instead (not tested, you should test it).

char * fgets ( char * str, int32_t num, File * stream ) {   int32_t i = 0;      while (i < (num - 1)) { // got room for newline and terminating null?     int16_t ch = stream->read();     if (ch < 0) // end of file       break;     str[i++] = ch;     if ('\n' == ch) // end of line       break;   }      if (i) { // room in buffer for terminating null     str[i] = 0;     return str;   }   else     return NULL; // buffer too small or immediate end of file }

Peter Olson

[shiftleftplusone]

@peabo7: thank you for your input, I'm just a beginner in using C, and hints a very much appreciated.

I have a question nevertheless:

You wrote "It has a buffer overrun because it doesn't pay attention to the num parameter." but I had coded in 1 line "while ( stream->available() && (i < num) ) " so num had been notified IMO very well - is that not true? or would it have to be replaced just by "while ( stream->available() && (i < (num-1) ) ) " instead ?

OTOH, getting "" returned instead of a NULL pointer was my intention because I supposed the handling of a NULL pointer in Sketch to be much more complicated IMO than just getting a nullstring "" instead. IMO "" could be tested very quickly instead of getting pointer address issues and being lead to Nirwana when processing the returned string instead further on (e.g., by itoa or sscanf or what ever). But I agree, in this specific case it had been called something slightly different, e.g. "fgets_" or something else.

[shiftleftplusone]

BTW, @devs: although peabo7's function mimics the "real" fgets() very well, this is just why I appreciated to have full stdio.h and stdlib.h support by Sketch instead of having to fake all these functions by one self with a risc of severe failures. So why having to invent the wheel from the scratch? It's all here, by gpp. (CMIIW) So my wish was to just provide full stdio.h and stdlib.h features, please!

[matthijskooijman]

I'm again confused. It seems that fgets is already available through libc (which is available and enabled by default):

matthijs@grubby:~$ grep fgets /usr/lib/avr/include/ -r
/usr/lib/avr/include/stdio.h:extern char        *fgets(char *__str, int __size, FILE *__stream);

The only thing that is not available, is default streams (stdin/stdout/stderr), since they make no sense on a microcontroller, or functions like fopen to get a FILE* pointing to a file, since there is no default filesystem.

I previously suggested, in another ticket, to use FDEV_SETUP_STREAM to create a custom FILE* that wraps Serial (or whatever Arduino Stream you want). Then you can just use the already available fgets, fprintf (and printf if you assign stdout).

Does this not satisfy your need? Do you perhaps not understand what I'm suggesting? Or are you of the opinion that such FILE*-creating code should be included with Arduino by default?

[shiftleftplusone]

2x yes ;) 1st, I do not understand your suggestion ;) 2nd, IMO SD files should be implemented to Sketch to work just the same as files usually work by C and the common stdio.h functions. So then we simply could use the common gpp stdio.h to work with Arduino SD files. 8-) (to be honest, I have no idea how this could be achieved) :-/

by SD, (myfile).open will be substituted by => => fopen (myFile).println => => fputs or fprintf (or what ever) (myFile).readln (what not exists) => => fgets and so on

[PaulStoffregen]

You really seem to have missed the concept that Arduino creates its own simplified APIs.

Arduino is designed to be a system that makes electronics easier for novices and hobbyists. These things are done differently from standard C for very good, very intentional reasons.

You're not the first person to come along and loudly insist Arduino should abandon its long-standing and very successful design, only because you personally want it to work the same way as other systems. You're not the first person to say Arduino's way is non-standard or technically inferior. Many others have come before you and made pretty much all these same arguments.

The obvious reality is Arduino is a very successful platform because is provides simplified APIs.

[shiftleftplusone]

I know that it IS different, but it's not the best way it is supposed to be. C is C is C is C. IMO Arduino is C and so it should work like C in the first line, probably providing some stuff which could make it (additionally) a little simpler here and there.

So first of all comes full C compatibility (like SNSI C99 plus stdio.h plus stdlib.h,

and then (maybe) comes some additional stuff.

[PaulStoffregen]

No, you do not understand.

Arduino is Arduino which uses C internally. It should work like Arduino.

[shiftleftplusone]

here in my topic about Sketch enhancements it is about making Arduino work with stdio.h functionality like C does by standard. So IMO (about which is the current topic) Arduino should be able to provide standard C compatibility.

Additionally of course it may provide additional features which makes it easier here and there to handle by hobbiists.

opening files by fopen is standard, so it should work with Arduino, same about fclose. fprintf is standard for writing data to file, so it should work for Arduino. fscanf is standard in C to read data from files, so it should work with Arduino.

If one likes to have addtionally SD-(myFile).println, why not? but that's secondary.

[PaulStoffregen]

No, it's is Arduino's primary API. You simply do not understand Arduino.

[shiftleftplusone]

I understand Arduino, but my topic is about enhancing Arduino by C compatibility. I think this is the point which YOU don't understand ? stdio.h is finally much more powerful than just file.write and file.read and file.println

[PaulStoffregen]

No, you obviously do not understand Arduino at all.

[shiftleftplusone]

well, maybe start reading at the TOP: I just don't want Arduino Sketch to stay as crippled as it currently is !!

having just SD-myFile.read to to read data from SD files ist simply rediculous !!

opening files by fopen is standard, so it should work with Arduino, same about fclose. fprintf and fputs are standard for writing data to file, so they should work for Arduino. fscanf and fgets are standard in C to read data from files, so they should work with Arduino, too.

If one likes to have addtionally SD-(myFile).println, why not? but that's secondary. - and I honestly don't see why myFile.println should be more convenient than fputs or fprintf ???

again: what I wish acc. to the TOP is Arduino to support and provide full stdio.h functionality, basicly, additionally to the existing API, not exclusively!

clearerr
fclose
feof
ferror
fflush
fgetc
fgetpos
fgets
fopen
fprintf
fputc
fputs
fread
freopen
fscanf
fseek
fsetpos
ftell
fwrite
getc
getchar
gets
perror
printf
putc
putchar
puts
remove
rename
rewind
scanf
setbuf
setvbuf
snprintf
sprintf
sscanf
tmpfile
tmpnam
ungetc
vfprintf
vfscanf
vprintf
vscanf
vsnprintf
vsprintf
vsscanf

objects:

stderr
stdin
stdout

types:

FILE
fpos_t
size_t

macro constants:

BUFSIZ
EOF
FILENAME_MAX
FOPEN_MAX
L_tmpnam
NULL
TMP_MAX

[mbanzi]

@PaulStoffregen gets Arduino :) thanks Paul

[mikaelpatel]

Is this a discussion about the Arduino API? I always thought the API was an attempt at OOP/C++. I would really like to discussion some of the constructs and the usage of classes. I do not see the elegance that @PaulStoffregen seems to see. What I see is something that would not give computer science students a degree let alone work in industry :) Let us use some standard industry metrics and evaluate the Arduino core and libraries before doing some of the claims above. @VogonJeltz gets software standards and reuse.

Cheers!

[peabo7]

On February 28, 2015 at 3:51 AM VogonJeltz notifications@github.com wrote:

thank you for your input, I'm just a beginner in using C, and hitns a very much appreciated.

I have a question nevertheless: You wrote "It has a buffer overrun because it doesn't pay attention to the num parameter." but I had coded in 1 line "while ( stream->available() && (i < num) ) "

so num had been notified IMO very well - is that not true?

Aha, I thought there was something funny about that line. The message displays differently in email from the list than it does when looking at it on Github. In email it says:

while ( stream->available() && (i str[i]=stream->read() ;

My apologies! Now I know to read on Github before replying. The list software is sensitive to text that could be construed as HTML, and the "< num" resembles a tag well enough to fool it.

Peter Olson

---

Reply to this email directly or view it on GitHub: https://github.com/arduino/Arduino/issues/2715#issuecomment-76517675

[PaulStoffregen]

@mikaelpatel - I would not use the word "elegance" to describe Arduino. Your Cosa system is the only hardware API I've ever seen that truly deserves to be called elegant. By comparison, Arduino is downright dirty. It's also technically limited in many ways, but powerful features and standards compliance are not Arduino's design goals...

Arduino is approachable and learn-able and usable for novices and absolute beginners. For anyone who wishes to argue otherwise, I can't hear you over the sound of Arduino's incredible success.

[shiftleftplusone]

to make my intention clear: i love the Arduino IDE because it's simple and handy and allows a very quick start for beginners, and I appreciate the convenient API for programming! I myself had never started with MCUs if it had not been possible as it turned out by the Arduino Sketch IDE.

But there are 2 points which have to be taken into account: 1st, not every Arduino user stays at Uno or Nanos but probably will proceed to Mega and Due and (hopefully soon) Tre for larger projects by higher demands 2nd, for higher demands one would appreciate to be no longer restricted to the Arduino Sketch world of "not quite" and "not exactly" (as once one John Hansen, the author of Lego Mindstorms IDEs apostrophied talking about his own NQC and NXC compilers) but might be able to use additionally either powerful functions of the real word of C which is no longer "not quite" or "not exactly" C!

Indeed, being tied to IDEs like Eclipse and committed to makefile and explicite static ar dynamic linking most Arduino beginners had become desperate and abandoned all attempts long before achieving the first simple program to run (just like me trying and failing by embedded C on an ARM 9 Linux system) - but when the first steps are done, and bigger projects start arising (e.g., for me: neural nets, astar, MonteCarlo filters, fast Fourier Transformation, speech recognition, autonomous path finding in real world environments) then more powerful programming facilities would be needed, even though still staying fine with the Arduino IDE!

(BTW, that's exactly why I'm eagerly waiting for the Tre feat. Sketch instead having to use the BBB or RasPi by their horrible OSs and PLs and IDEs instead, and I must admit: I hate OOP like the devil hates the Holy Water!!).

Now do you see my point?

So yes, I daresay : I do understand Arduino, I love it and I'm glad to be able to use it, but I want some elaborated, more advanced features ADDITIONALLY, e.g. stdio.h features.

Would that be OK?

[mikaelpatel]

@PaulStoffregen Arduino is a great success due to (in my opinion) the needs for cheap hardware for teaching and hobby projects. It is not really Arduino per se but the trend towards "nerds" becoming respectable and that technology is now back on the curriculum (at least here in Sweden).

Arduino could be even more successful with component based hardware AND software. There are a lot of software professionals that are willing to give a helping hand. Cosa is a demonstration of applied theory and methodology from over 10 years of teaching Computer Science at University and over 20 years experience of large scale embedded systems projects at one of the worlds largest Telecom companies. Cosa is in no way a product - so I am cheating and only halfway there.

We could discuss how to add support for larger projects and for those who have come past being beginners. In that case I think the discussion will actually end up in software standards and being able to share software across several platforms. Software development is the largest cost in any project.

Already with the Arduino Mega the lack of support for larger projects became obvious. The whole SPI discussion is a good example. And we are not really done with that as there is a level missing when we go for multi-tasking.

Cheers!

[shiftleftplusone]

YES, defintely YES: multitasking (i.e.: preemptive multitasking - maybe something simple, similar like featured by POSIX pthread, please no overcomplicated OSEK) really IS an another issue!!

BUT PLEASE NO COMPULSATORY OOP! PLEASE STAY WITH THE SIMPLICITY OF STRAIGHT PROCEDURAL PROGRAMMING AND THE SIMPLE IDE !

But for the first step, acc. to the topic of this thread and my TOP: please provide some additional stdio.h functions additionally for the awsome Arduino IDE ! (and then also preemptive multitasking).

[PaulStoffregen]

@mikaelpatel Yes, I agree, much needs to be done to facilitate larger projects. This particular thread probably isn't a very productive way to talk of such things.

[shiftleftplusone]

supporting standard C syntax and libs is NOT productive for larger projects ?? Are you SERIOUS ???

Just the contrary! It's the mandatory first basic step !!

[mikaelpatel]

@VogonJeltz Back to the original issue (sorry for the diversion). Did you see this https://github.com/greiman/SdFat/tree/master/SdFat?

Bill has actually taken the time to implement the ios standard (https://github.com/greiman/SdFat/blob/master/SdFat/ios.h) for SD as far as I can tell. But the necessary parts for the AVR GCC stdio is not very hard and I agree should be implemented. If it should be in the Arduino core might be another issue as that is the product management dimension.

The current Serial/Print/etc interface in Arduino could be improved to put it mildly. Compare for instance Cosa UART with Arduino Serial. Why not include an IOBuffer class from the start? Ring buffers are always needed. And why not make that implement a device interface?

Last @PaulStoffregen, we might need to bring up the issue on the developers group? I would at least like to start discussing the need for back-end interfaces so that device drivers for the different bus protocols can be improved (e.g. I2C, SPI, 1-Wire). The Cosa code is full of delegation to interface such as the IOStream::Device to achieve this structuring of code. This has started to scale nicely. And actually a C++ variant of the different device driver structures in Linux.

Cheers!

[shiftleftplusone]

Mikael, thank you for your input! Unfortunately I'm not used to github information organization and so I don't actually see what you mean. And I'm just a beginner with Arduino, I have started with it just a couple of months ago, so I'm not very well informed about all of the implementations and classes etc..

But as a workaround, I meanwhile have written my own additional function (with a little help from my friends), which mimics fprintf() sufficiently and so it's a substitute FTM. (Nevertheless, a complete support of stdio.h would be still appreciated!)

//*****​****

include < SPI.h>

include < SD.h>

include < stdarg.h>

include < stdio.h>

int fprintf_ ( File * stream, const char fmtstr[], ... ) { char str[1024]; va_list arguments; int16_t num;

va_start( arguments, fmtstr); num = vsprintf(str, fmtstr, arguments); // #debug // Serial.println( "string to write to file="); Serial.println( str);
stream->print(str); va_end( arguments );

return num; } //*****​****

// call, e.g.:

File myFile; int16_t num=20;

fprintf(&myFile, fmtstr, ...); // .. fgets ( sdata, num, &myFile );

[peabo7]

You can get a little extra safety by changing

num = vsprintf(str, fmtstr, arguments);

to

num = vsnprintf(str, sizeof str, fmtstr, arguments);

Peter Olson

[shiftleftplusone]

thank you very much, I will do so! Finally things like these will enable me writing portable code which I will be able to utilize even later on different platforms! (I already have tons of code written in a "not-quite-C" dialect which is now completely useless to me for Arduino, and I swore I will never make this mistake again. For the future I swore Better no code at all than non-compatible C-(un)like code!! ) So just again to the devs:

please make SD files compatible to C files in order to utilize stdio.h WHICH IS FINALLY ALREADY HERE IN THE ARDUINO SYSTEM!

[mikaelpatel]

@VogonJeltz Did you see this? http://www.nongnu.org/avr-libc/user-manual/group__avr__stdio.html and http://www.nongnu.org/avr-libc/user-manual/group__avr__stdio.html#gaea2b6be92ead4673bc487b271b7227fb.

Checkout the examples on how to bind a device to your own FILE. It is a matter of a few lines of code to bind the Arduino Serial or SD to the AVR stdio FILE and start using that.

Remember this is AVR GCC stuff and not Arduino. If you consider that to be wrong you have to go to that right forum.

Cheers!

[shiftleftplusone]

no, sorry, as I already wrote: I don't understand where to look and what to change. I don't get clear with the github maze and the lot of code you linked to. Can you just write down the1 code line I have to write in order to make an Arduino SD File a FILE * ? So having written this 1 line subsequently being able to use all the already existing stdio.h functions and features?

Additionally, I'm also working (mostly) with my Arduino Dues (ARMs) and most likely with a new PCduino which are not AVR any longer.

Otherwise I would prefer it to be available ready, finished, out of the (Arduino-) box, for Mega, Due, PCduino, and the same code for the Tre (hopefully coming soon) what would make no difference for the platforms if all and everything simply will be just standard C.

[mikaelpatel]

That will not happen :)

[shiftleftplusone]

What will not happen ?

[mikaelpatel]

"...it to be available ready, finished, out of the (Arduino-) box, for Mega, Due, PCduino, and the same code for the Tre..." will not happen any time soon :)

[shiftleftplusone]

But that's exactly what my (this) topic is about! What part didn't you understand about " I appreciated to have full stdio.h and stdlib.h support by Sketch instead of having to fake all these functions by one self " ? stdio.h is already here, nothing had to be faked (and actually, I don't want anything faked!) ! why not a have files on SD which work like files on embedded ARM-Linux MCUs or on a Windows PC? Every standard C function would suddenly work fine then, automatically, on either platform !

[mikaelpatel]

@VogonJeltz I do not represent Arduino hardware or software. I was just trying to help you use AVR stdio and how you can bind functions to that to solve your issue. This is as far as I will go. Hope you get this issue solved. Cheers!

[shiftleftplusone]

well, I don't want a substitute of faked functions. Some of these functions I already faked by my own, but that doesn't make e.g. fprintf() work with SD files, it just makes a fprintf_ faked function work with SD cards.

I want the original stdio.h to work with the files and everything on Arduino just as if they were real standard C stuff.

Standard gpp C inclusively stdio.h are already integrated here. So where is the problem?

[PaulStoffregen]

The problem is with you, and your lack of understanding of what Arduino is.

[shiftleftplusone]

why are you offending and insulting Arduino users by unqualified arrogant verbal abuse? who do you think you are?!

[PaulStoffregen]

Before you label me arrogant, perhaps you should consider who is demanding an extremely successful open source project, used by millions of people, must change its design to meet his desires?

Before you cast yourself as the victim of verbal abuse, perhaps you could re-read some of the very angry message you've written here?

But most importantly, perhaps you should try to understand Arduino's design and goals?

[PaulStoffregen]

Regarding who is who, I have personally fixed dozens of bugs and contributed numerous features to Arduino over the last 6 years.

What have you accomplished?

[shiftleftplusone]

let's start with you - are you working for the Arduino company or are you just an arrogant nerd and bigmouth? Or both ?

[mikaelpatel]

@VogonJeltz It took me ten minutes to fix the issue by add the necessary lines to get SD working with the AVR GCC stdio.h function. Unfortunately I would never share that with you - ever. Please close this issue and read more Vogon poetry :) - you actually got your name right :)

[shiftleftplusone]

please keep it clean on the subject

tell me please what you did to make SD work with stdio.h (I don't want it to work with a different lib, but with the original gpp lib which I already included for faking fprinf). I also don't see what exactly the AVR lib is for - why is it called AVR lib ? will it work for ARM as well ? Or just for AVRs ?

fprintf is already existing in stdio.h coming with Sketch, but it doesn't work with SD File. If I call this function I get a compiler error - not about an undefined fprintf function, but about

This is what I'm talking about: making the functions available which already exist in the stdio.h coming with the IDE.

ps,
I guess the problem is that Sketch maybe has no FILE \*  implementation as it's usual.
So this would mean to first of all implement FILE \*  to Arduino Sketch, additionally to the existing API.