Reading tab-delimited datafiles

[rtoy]

Imported from SourceForge on 2024-07-08 11:05:51 Created by sikkenogetvrovl on 2024-05-26 17:15:55 Original: https://sourceforge.net/p/maxima/bugs/4306

---

I often use maxima/wxMaxima to read tab-delimited data files and is has been working like charm when using

Stream : openr(directory(DataFile[1])[1]);
" some loop :" ;
dataLine : readline(Stream);
- "data-handling code"

However during some of the latest Nightly Builds the readline() noes not read the TAB-character anymore :-(

I has to revert back to a old wxMaxima build I have on my computer from 2024/01/30 to get it to work

The next wxMaxima build, I have stored on my computer, is 2024/03/20, but this one and later builds contains the above failure/change.

I would be very pleased if you can help !

Runs wxMaxima on windows 11 / 64 bit

---

Instead of using some of the build-in functions, I have to use my own programming to read the data files as they contain data in a mixed way:

/* Length=43.10 */
/* IntTime=30000 */
/* StrobeDelay=10000 */
/* Lamppower=1 */
/* Navg=33 */
/* UTCTime=20240507-143923 */
/* Latitude=nan */
/* Longitude=nan */
/* Speed=nan */
/* Heading=nan */
/* Temp=31.812000 */
189.182373 {tab} 0.427444
189.313156 {tab} 0.427444
189.443924 {tab} 0.427444
.
.

---

I have also posted my question here: https://github.com/wxMaxima-developers/wxmaxima/issues/1920 But this forum might be more relevant.

[rtoy]

Imported from SourceForge on 2024-07-08 11:05:52 Created by robert_dodier on 2024-05-26 17:25:28 Original: https://sourceforge.net/p/maxima/bugs/4306/#c3c1

---

Rene, thanks for the bug report. What is the error message which you are encountering?

You mentioned wxMaxima; does the operation work as expected if you run it in command line Maxima instead?

[rtoy]

Imported from SourceForge on 2024-07-08 11:05:56 Created by sikkenogetvrovl on 2024-05-26 21:00:14 Original: https://sourceforge.net/p/maxima/bugs/4306/#08b9

---

Hi Robert There is no error message, the readline() reads the lines from my csv-file without any complains. However when the data in the csv-file are separated by tabs = ascii(11) the readline() removes the tab character. For example : If a line in the csv is 189.443924 {tab} 0.427444 readline reads it as 189.4439240.427444 which makes it impossible to separate the two numbers.

Previously this as worked like a charm - readline() reads 189.443924 {tab} 0.427444 so I could use split( dataLine, ascii(11) , true ) ) to get the two numbers

I don't think is has anything to do with the wxMaxima interface - the functionality of readline() seems to have been changed in maxima during the past couple of months :-(

As I wrote above, my data files are somewhat special as they contain data in two ways. Therefore I has to write my own code to read out the data.

I can not change the data files to be able to use readmatrix() or like as I have thousands of them ....

[rtoy]

Imported from SourceForge on 2024-07-08 11:05:59 Created by robert_dodier on 2024-05-27 20:04:54 Original: https://sourceforge.net/p/maxima/bugs/4306/#ae84

---

Rene, looks like there are a few different things going on.

First, the ASCII tab character is control-I, and since I is the 9th character, tab is ascii(9), not ascii(11) (unless you have modified the global variable ibase to allow octal number input via ibase: 8.; note that the trailing dot with no further digits indicates a decimal integer, whatever the value of ibase). If you try split(dataLine, ascii(9), true), does it work as expected?

Second, can you specifically try the same operation in command line Maxima to see if it works or not? I guess you will have to be careful to execute command line Maxima from the same installation which has the non-working wxMaxima since you have multiple installations.

Third, can I ask you to try map(cint, charlist(dataLine)) in the non-working wxMaxima version, and report the output here? (cint returns the integer code for a character and charlist returns a string split into a list of 1-character strings.)

I've reviewed the commit logs for both Maxima and wxMaxima from Jan 1 of this year to the present, and I don't see any changes which indicate changes in the treatment of tab characters or stream input in general. There could still be some change of which I haven't understood the implications, but anyway, nothing looks suspicious.

I tried reading a file containing tab characters using the current version of Maxima + SBCL on Linux, and the tabs are apparently read successfully. The first line in the file is aaa<tab>bbb:

(%i2) S: openr ("tabbed.data") $
(%i3) L1: readline (S);
(%o3)                        aaa    bbb
(%i4) map(cint, charlist(L1));
(%o4)              [97, 97, 97, 9, 98, 98, 98]

[rtoy]

Imported from SourceForge on 2024-07-08 11:06:03 Created by sikkenogetvrovl on 2024-05-28 09:17:30 Original: https://sourceforge.net/p/maxima/bugs/4306/#b1ec

---

Arghh... my colleague has apparently decided to change "Vertical tab" (ascii 11) with "Horizontal tab" (ascii 9) in the data files

Also, the previous versions of maxima/wxMaxima has chaged the display of the string from:

"Old" maxima/wxMaxima:

headerLine : readline(Stream); "189.184875�22.000000"

map(cint, charlist(headerLine)); [49,56,57,46,49,56,52,56,55,53,9,50,50,46,48,48,48,48,48,48,13]

"New maxima/wxMaxima :

headerLine : readline(Stream); "189.18487522.000000"

map(cint, charlist(headerLine)); [49,56,57,46,49,56,52,56,55,53,9,50,50,46,48,48,48,48,48,48,13]

I did not previously include ascii 9 handling in my code together with wxMaxima not displaying the tab (189.184875�22.000000) anymore - that what did confuse me :-( Thanks for reminding me on map(cint, charlist(headerLine)) !!

I apologize for bothering you ...

[rtoy]

Imported from SourceForge on 2024-07-08 11:06:06 Created by robert_dodier on 2024-05-28 16:16:11 Original: https://sourceforge.net/p/maxima/bugs/4306/#57c9

---

Hi Rene, thanks for the update. From what you said, the problem is the presence of ascii(9) versus ascii(11), so it seems like the problem is not in Maxima itself. I'm marking this report as "not a bug" accordingly. No problem, please open a bug report or post a message to maxima-discuss whenever you run into something. Hope this helps, and all the best.

[rtoy]

Imported from SourceForge on 2024-07-08 11:06:10 Created by sikkenogetvrovl on 2024-05-28 19:22:40 Original: https://sourceforge.net/p/maxima/bugs/4306/#4205

---

Hi Robert Yes, it was not a bug or change in maxima. The only change is wxMaxima not displaying ascii(9) ascii(11) etc anymore - but that it another forum.. Thanks for your time/help. /Rene'