asmoses uses only first 20 rows to determine data type

singnet / asmoses

Port of MOSES (https://github.com/opencog/moses) for the AtomSpace (https://github.com/opencog/atomspace)

Other

6 stars 5 forks source link

asmoses uses only first 20 rows to determine data type #144

Open noskill opened 4 years ago

noskill commented 4 years ago

I guess valid fix would be reading all the data or providing data types in separate csv file. @ngeiswei what do you think?

https://github.com/singnet/asmoses/blob/2d7fe0c65a3d00138e864ef8ee848e266eb6b7b2/opencog/data/table/table_io.cc#L916-L918

noskill commented 4 years ago

also, the comment is misleading, the negative number doesn't mean reading all lines

ngeiswei commented 4 years ago

I would suggest introducing yet another user option to set maxline, as well as supporting negative numbers.

ngeiswei commented 4 years ago

I don't mind if the default is negative (reading all lines) but I do recall that systematically reading all lines was incurring too much setup overhead, which is why I suggest adding a user option.