Addind Debug Mode to Matrix Export

[msdogan]

@jrmerz Similar to debug mode in HEC-PRM run from HOBBES network with a command prm build --prefix test --debug All, I see a great value on adding DEBUG MODE to matrix export. That way, we should be able to identify issues all at ones.

[jrmerz]

@msdogan ok, I have to admit some ignorance with the --debug flag. So for debug --ALL you would like to see something like:

SR_ISB.2002-08-31,DBUGSNK.2002-08-31,0,0,1,0,1000000 DBUGSRC.2002-08-31,SR_ISB.2002-08-31,0,0,1,0,1000000

? If so, is 1 for amplitude correct?

[msdogan]

@jrmerz yes, this is correct. k=0, cost=0, a=1, lb=0, ub=1000000. And also links SOURCE DBUGSNK.2002-08-31,0,2000000,1,0,1000000 and DBUGSNK.2002-08-31 SINK,0,2000000,1,0,1000000 These are source and sink links that we apply very high penalty. But, we apply high penalties only on these links. SR_ISB.2002-08-31,DBUGSNK.2002-08-31,0,0,1,0,1000000 and DBUGSRC.2002-08-31,SR_ISB.2002-08-31,0,0,1,0,1000000 will have zero cost as you said.

[msdogan]

There is also some other issues, for example there is a inflow to D43a (https://cwn.casil.ucdavis.edu/#info/D43a), but it does not show up in the exported matrix. I couldn't find a reason for that. One drawback right now is that, Pyomo shows one error at a time. When it finds an error, it stops. Maybe, I should wait until Debug flag to attack all these issues.

[jrmerz]

Ya, about to have that flag ready for you. If you don't think the missing D43a is part of this issue, go ahead a create a new one.

[msdogan]

Great! No missing D43 inflow is not part of this issue. I just wanted to say :) So, new calvin-network-tools version for it? 1.0.38?

[jrmerz]

Just published. --debug is available for the matrix @ v1.0.38

[msdogan]

This is working perfectly. However, could you please delete time information on SOURCE for SOURCE.2002-08-31 DBUGSRC.2002-08-31 0 2000000 1 0 10000000 and SOURCE.2002-09-30 DBUGSRC.2002-09-30 0 2000000 1 0 10000000

So, correct representation would be: SOURCE DBUGSRC.2002-08-31 0 2000000 1 0 10000000 and SOURCE DBUGSRC.2002-09-30 0 2000000 1 0 10000000

Thanks

[jrmerz]

Ok, date is removed from SOURCE @ v1.0.39

[msdogan]

@jrmerz okay, this is working. Debug mode will save a lot of time. Thanks.

Ooops. I got a FATAL ERROR when I tried to export all 82-year period in debug mode with cnf matrix --format csv --start 1921-10 --stop 2003-09 --ts . --to networklinks --max-ub 10000000 --outnodes networknodes --debug All

[jrmerz]

ok, ya, this will take a bit. Going to have to request more memory at runtime, but even then going to hit some other walls. I'll let you know when I have fixes.

[jrmerz]

@msdogan give it a go. I fixed memory issue on linux and mac. Should work on windows, but let me know.

[msdogan]

@jrmerz this exports the 82-year matrix to .csv but it is giant (300MB). And I can't even open it. It says "File not loaded completely" I wonder if there is a way to reduce the size. Maybe we can do that right now, there is DBUGSRC and DBUGSINK nodes for each time step, DBUGSRC.2002-09-30, DBUGSNK.2002-09-30. If we create only one DBUGSRC and DBUGSINK without time-step information, and connect nodes to those, we can eliminate 2000 links. What do you think?

[msdogan]

@jrmerz Update: let's leave time-step information there, otherwise I would need to update post-processors. Besides, 2000 links is nothing compared to the rest.

[msdogan]

@jrmerz can I close the issue?

[jrmerz]

Ya, there are current 5,185,762 so removing those would be a 0.03% file reduction.

[jrmerz]

I would recommend running with a sub-network if you want to run over 82 year period.

[jrmerz]

And sure if you think it's go to go.

[msdogan]

yeah it is what it is. 82-year run file w/o debug mode is also big. About 200MB

[jrmerz]

FYI, bash tools like less, more and tail should let you inspect the file. They should be available on windoz via the git-bash command prompt.