search

gsimardnet / PowerQueryNet

Run M Language (Power Query Formula Language) from anywhere.
MIT License
189 stars 32 forks source link

Output CSV file performance? #9

Closed hiptopjones closed 5 years ago

hiptopjones commented 5 years ago

I have a query that reads records from a CSV file, performs some transformations, and outputs to another CSV file. The output file has about 35,000 rows.

If I run it with CSV output parameters, it takes 8.5 minutes:

pqnet query.pq -c credentials.xml -o csv -f output.csv

If I run it with no output parameters, it shows up in the built-in GUI and takes less than 5 seconds. I can select all and copy all 35,000 rows from that GUI and paste it into a text editor in a total of 10 seconds, so it's not due to lazy evaluation.

pqnet query.pq -c credentials.xml

Why does the formatting and output to CSV take so long? Am I doing something wrong?

Thanks!

gsimardnet commented 5 years ago

I have made some tests and noticed the same problem.

By doing some other tests I found a quick fix and I just released a new version that will improve the performance drastically. Please download Version 1.1.1 and test again.

Thanks for raising that issue.

hiptopjones commented 5 years ago

Thanks for the quick investigation and fix! With the new version, the output CSV file is written in 3-5 seconds, which is awesome.