search

NREL / rplexos

Other
18 stars 13 forks source link

Problem processing a large solution file (.zip) #56

Open jhoncr opened 7 years ago

jhoncr commented 7 years ago

process_solution(location_solution) rplexos debug: - Solution: 'results.zip' rplexos debug: Reading XML file and saving content rplexos debug: PLEXOS version: '7.400 R01 Gold ' rplexos debug: rplexos version: '1.1.10' rplexos debug: Adding extra tables to the database rplexos debug: t_key has 462844 rows rplexos debug: temp_key has 462844 rows rplexos debug: Creating final database and adding basic data rplexos debug: Creating data tables and views rplexos debug: Reading interval binary data rplexos debug: Max position: 1785730104 rplexos debug: Adjusted max position: 1785730104 rplexos debug: Sum of length: 1785822360 rplexos debug: Sum of adjusted length: 1785822360 Warning: Problem with length of 't_key_index' for period 0 in file 'results-temp.db' *** rplexos debug: 1785729360 values read Error: Problem reading interval binary data (reached end of file). 8760 values requested, 3360 returned.

danielsjf commented 7 years ago

Yes I had this problem as well for some DBs. Not sure yet what the problem might be. Could you share the db? I cannot guarantee that I can solve it though. But if you want to investigate it yourself, I can always accept a pull request.

danielsjf commented 7 years ago

Could also be related to #27 if you're on a Unix system.

jhoncr commented 7 years ago

Ok, I'm running it on a Win64 environment. Where is the warning coming from? Do you have the same warning proceeding the same problem?

danielsjf commented 7 years ago

The error message is generated from the code here: https://github.com/NREL/rplexos/blob/de84e8bd90b32b88b2b1b59ac7ecdde4dd9079c6/R/process_solution.R#L272

I am not the original developer of the code, but this is how I understand it. In the Plexos solution.zip, you have a few files. One is an xml, the others are binary files for the various aggregations (interval, day, week, month, year,...). Rplexos first reads the xml. This xml also contains a mapping for the binary files. The binary files are read in different chunks (and immediately written to the sqlite db). These chunks are identified with the time, collection and item (e.g. data_interval_generator_generation) and each chunk forms one table in the final sqlite db.

What is basically happening here is that rplexos tries to read a chunk of the binary file (which should be in that position according to the xml), but then the file ends before the full chunk was read. This could be due to a corrupt Plexos solution file or an edge case in which there is a bug in rplexos.

Unfortunately, I don't have a db anymore that produces the error. It was a former colleague that encountered the problem.