usc-isi-i2 / dig-etl-engine

Download DIG to run on your laptop or server.
http://usc-isi-i2.github.io/dig/
MIT License
101 stars 39 forks source link

Timeseries annotation: extractSpreadsheet very slow in col-based sheets #187

Open majidghgol opened 6 years ago

majidghgol commented 6 years ago

The sample file for Earthquakes data (issue 175) has about 8000 rows and 20 columns. I introduced a very simple mapping spec to extract only the magnitude of the earthquakes (one column). However, I noticed every iteration in parse_col_ts takes 1s, which makes it need more than 2 hours to finish. I am not sure if it is specific to my machine, but as far as I know pyexcel is much more efficient in row by row reading of excel file. So, it might be due the fact that the spreadsheet is read col by col in this function.

The mapping spec I am using:

[
    {
        "TimeSeriesRegions":
        [
            {
                "orientation": "col",
                "locs": "[2:8498]",
                "cols": "[E:E]",
                "metadata": [
                    {
                        "orientation": "row",
                        "name": "data_label",
                        "loc": 1
                    }
                ],
                "times": {
                    "locs": "[A]"
                }
            }
        ],
        "Metadata":
        [

        ]
    }
]