error reading values like "5.07856133975082E-4" from files

[GoogleCodeExporter]

What steps will reproduce the problem?
1. generate (vector) sequences
2. write sequences to file
3. read sequences from file

What is the expected output? What do you see instead?
reading an vector sequence works fine in general. but when there is a
vector like "[ 5.07856133975082E-4 2.221522656000841 ];" to read, the
system tells you "Line 49: Number or ']' expected" since the parsing
function checks for beeing a number. ("E" and "-" are no numbers :) )

What version of the product are you using? On what operating system?
ObservationVectorReader.java - v.0.6.1

Please provide any additional information below.
"new MarkovGenerator<ObservationVector>(hmm).observationSequence(length)"
SOMETIMES returns a value like "5.07856133975082E-4" which cant be read
from a file later. so the makrovGenerator should stop at a certain length
of number or (better): reading those numbers with "E-x" should be fixed...

kind regards,
Ben

Original issue reported on code.google.com by vamos.be...@gmail.com on 15 May 2010 at 6:46

[GoogleCodeExporter]

this is really a plague, especially if you're trying to train 'real' models 
where transitions probabilities are potentially very low (in which case, just 
hacking the Opdf write methods to add more decimals doesn't make any sense...)

Pick the attached class, place it in the 'io' directory and replace all 
references to StreamTokenizer for CustomStreamTokenizer.  This attached class 
is a simple modification of the original Sun StreamTokenizer implementation: it 
now supports the scientific notation for the 'number' class.

Original comment by jean.phi...@gmail.com on 12 Jul 2011 at 8:28

Attachments:

• CustomStreamTokenizer.java