rocketlaunchr / dataframe-go

DataFrames for Go: For statistics, machine-learning, and data manipulation/exploration
Other
1.16k stars 93 forks source link

Allow converter types on csv import #54

Closed jdfergason closed 3 years ago

jdfergason commented 3 years ago

On CSV import allow native series for float64, int64, and string when a Converter is used.

pjebs commented 3 years ago

Can you tell me why you aren't just directly using DictateDataType instead?

jdfergason commented 3 years ago

I am using DictateDataType, but I need custom converters because my CSV file has some null values that are indicated with a '.'.

Here's my code:

DictateDataType: map[string]interface{}{
        DateIdx: imports.Converter{
            ConcreteType: time.Time{},
            ConverterFunc: func(in interface{}) (interface{}, error) {
                return time.Parse("2006-01-02", in.(string))
            },
        },
        symbol: imports.Converter{
            ConcreteType: float64(0),
            ConverterFunc: func(in interface{}) (interface{}, error) {
                v, err := strconv.ParseFloat(in.(string), 64)
                if err != nil {
                    return math.NaN(), nil
                }
                return v, nil
            },
        },
    },

My suggested changes also make the API more consistent. It's confusing to get a SeriesTime as expected but then have the float64 field return SeriesGeneric if using the imports.Converter.

pjebs commented 3 years ago

Does NilValue option fulfill your needs?

jdfergason commented 3 years ago

for this specific use case yes, it would.