Representative periods ordering and mapping settings

RowanPeng commented 1 year ago

Hi,

Thanks for sharing this spineperiods template.

I have a question about the way to map the representative days with the non-representative days. At this stage, I was doing this steps manually. I was wondering whether there are any other autimatical ways to do this.

Many thanks,

Peng

RowanPeng commented 1 year ago

For this stage, I integrated the representative_periods in the model. I was trying to find the representative days and orders in Julia. However, I met this problem. Do you any ideas about how to solve this?

Plus: I just installed and updated the version of HiGHS.

Many thanks,

Peng

RowanPeng commented 1 year ago

Hi @DillonJ ,

Thanks for your time and guidance today. I have tried to update the SpineOpt as well. But it still shows like this. It says that I don't have the SpineOpt dependency in SpinePeriod package. I was wondering whether you have any ideas about this.

I look forward to your reply.

Many thanks,

Yuwen Peng

RowanPeng commented 1 year ago

And also I tried to add the dependencies "SpineOpt" manually in "SpinePeriods" in the Manifest documents. It doesn't work.

Regards,

Yuwen Peng

RowanPeng commented 1 year ago

And when I tried to put Pkg.precompile() it shows like this.

julia> Pkg.precompile()
Precompiling project...
✗ BinaryProvider
✗ PyCall
✗ CPLEX
✗ Clp
✗ SpineModel
✗ SpinePeriods
0 dependencies successfully precompiled in 26 seconds. 145 already precompiled.

ERROR: The following 3 direct dependencies failed to precompile:

CPLEX [a076750e-1247-5638-91d2-ce28b192dca0]

Failed to precompile CPLEX [a076750e-1247-5638-91d2-ce28b192dca0] to "C:\Users\Jeremy Peng\.julia\compiled\v1.9\CPLEX\jl_12C1.tmp".
ERROR: LoadError: CPLEX not properly installed. Please run Pkg.build("CPLEX")
Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] top-level scope
@ C:\Users\Jeremy Peng.julia\packages\CPLEX\rmvSt\src\CPLEX.jl:7
[3] include
@ .\Base.jl:457 [inlined]
[4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base .\loading.jl:2045
[5] top-level scope
@ stdin:3
in expression starting at C:\Users\Jeremy Peng.julia\packages\CPLEX\rmvSt\src\CPLEX.jl:1
in expression starting at stdin:3

Clp [e2554f3b-3117-50c0-817c-e040a3ddf72d]

Failed to precompile Clp [e2554f3b-3117-50c0-817c-e040a3ddf72d] to "C:\Users\Jeremy Peng\.julia\compiled\v1.9\Clp\jl_212B.tmp".
ERROR: LoadError: You have installed version 1.17.7 of Clp, which is not supported by Clp.jl. If the version change was breaking, changes will need to be made to the Julia code. Please open an issue at https://github.com/jump-dev/Clp.jl.
Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] top-level scope
@ C:\Users\Jeremy Peng.julia\packages\Clp\nnuI6\src\Clp.jl:24
[3] include
@ .\Base.jl:457 [inlined]
[4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base .\loading.jl:2045
[5] top-level scope
@ stdin:3
in expression starting at C:\Users\Jeremy Peng.julia\packages\Clp\nnuI6\src\Clp.jl:1
in expression starting at stdin:3

SpinePeriods [163584a9-52e1-4a4e-9262-a8c2d5456af4]

Failed to precompile SpinePeriods [163584a9-52e1-4a4e-9262-a8c2d5456af4] to "C:\Users\Jeremy Peng\.julia\compiled\v1.9\SpinePeriods\jl_66C6.tmp".
ERROR: LoadError: ArgumentError: Package SpinePeriods does not have SpineOpt in its dependencies:

You may have a partially installed environment. Try Pkg.instantiate()
to ensure all packages in the environment are installed.
Or, if you have SpinePeriods checked out for development and have
added SpineOpt as a dependency but haven't updated your primary
environment's manifest file, try Pkg.resolve().
Otherwise you may need to report an issue with SpinePeriods
Stacktrace:
[1] macro expansion
@ .\loading.jl:1634 [inlined]
[2] macro expansion
@ .\lock.jl:267 [inlined]
[3] require(into::Module, mod::Symbol)
@ Base .\loading.jl:1611
[4] include
@ .\Base.jl:457 [inlined]
[5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base .\loading.jl:2045
[6] top-level scope
@ stdin:3
in expression starting at C:\Users\Jeremy Peng.julia\packages\SpinePeriods\4TN6Z\src\SpinePeriods.jl:20
in expression starting at stdin:3
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\Types.jl:69
[2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\API.jl:1581
[3] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\API.jl:156
[4] precompile(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\API.jl:145
[5] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\API.jl:171
[6] precompile()
@ Pkg.API C:\Users\Jeremy Peng\AppData\Local\Programs\Julia-1.9.1\share\julia\stdlib\v1.9\Pkg\src\API.jl:162
[7] top-level scope
@ REPL[9]:1

Regards,

Peng

Tasqu commented 1 year ago

And also I tried to add the dependencies "SpineOpt" manually in "SpinePeriods" in the Manifest documents. It doesn't work.

Regards,

Yuwen Peng

It is not recommended to edit the Manifest.toml manually (see https://pkgdocs.julialang.org/v1/toml-files/). As far as I can tell from the above, SpinePeriods doesn't seem to be installed correctly. If you simply want to use SpinePeriods.jl, the easiest way to install it is from the SpineJuliaRegistry (see the README.md):

using Pkg
pkg"registry add https://github.com/Spine-tools/SpineJuliaRegistry"
pkg"add SpinePeriods"

If you need to modify SpinePeriods and install it locally (using the Pkg.develop(<path_to_spine_periods>)), you need to set up the module yourself. This is easiest done by opening Julia in the root folder of SpinePeriods (the one with Project.toml) and performing the following steps:

julia> ]
(@vX.X) pkg> activate .
(SpinePeriods) pkg> instantiate

Or if you prefer working using Pkg (e.g. in older Julia versions):

using Pkg
Pkg.activate(".")
Pkg.instantiate()

This should automatically install the dependencies of SpinePeriods. Note that if you want to use local SpineOpt or SpineInterface versions as well, you will need to install them locally in the SpinePeriods environment using the Pkg.develop command:

julia> ]
(@vX.X) pkg> activate .
(SpinePeriods) pkg> develop <path_to_local_spine_interface>
(SpinePeriods) pkg> develop <path_to_local_spine_opt>

Or using Pkg:

using Pkg
Pkg.activate(".")
Pkg.develop("<path_to_local_spine_interface>")
Pkg.develop("<path_to_local_spine_opt>")

RowanPeng commented 1 year ago

Hi @Tasqu

Thank you for your reply and explanation. I think I used to add the "Spineperiods" packages from SpinePeriodRegistry. And I tried it again today, it showed me that no changes in the Project.toml and Manifest.toml.

julia> using Pkg

julia> pkg"registry add https://github.com/Spine-tools/SpineJuliaRegistry"
     Cloning registry from "https://github.com/Spine-tools/SpineJuliaRegistry"
Registry `SpineRegistry` already exists in `C:\Users\Jeremy Peng\.julia\registries\SpineRegistry`.

julia> pkg"add SpinePeriods"
   Resolving package versions...
  No Changes to `C:\Users\Jeremy Peng\Project.toml`
  No Changes to `C:\Users\Jeremy Peng\Manifest.toml`

I looked at the messages it showed me, I noticed that the SpinePeriods package was added in this path rather than some folder in .Julia. Is that correct?

Regards,

Peng

DillonJ commented 1 year ago

Hi @RowanPeng. Would it make sense to uninstall SpinePeriods ( Pkg.rm("SpinePeriods") and then try again, installed from sources - clone the SpinePeriods repo locally and "develop" it as per @Tasqu's instructions - and then post back here with your experience

RowanPeng commented 1 year ago

Hi @DillonJ and @Tasqu . Thank you for your help and suggestions.

I remove SpinePeriods package, and then cloned to my local device. Here, I would like to use the develop command. However, it appeared to be like this.

julia> using Pkg

julia> pkg"dev <C:\Users\Jeremy Peng\SpinePeriods.jl>"
ERROR: `<C:\Users\Jeremy` appears to be a local path, but directory does not exist

However, the git cloned repository did exist in this path. Do you have any ideas about this? To be honest, this was the same situation when I added SpineOpt and SpineInterface from sources.

Regards,

Peng

DillonJ commented 1 year ago

Hi @RowanPeng your syntax looks wrong. You might want to look at basic Pkg usage here: https://docs.julialang.org/en/v1/stdlib/Pkg/

From the julia prompt it is:

Pkg.develop("C:\Users\Jeremy Peng\SpinePeriods.jl")

From the pkg prompt it is:

dev "C:\Users\Jeremy Peng\SpinePeriods.jl"

Either way, it looks like you have spurios "<" and ">" characters in your path

RowanPeng commented 1 year ago

Hi @DillonJ thank you for your correction. After uninstalling and then adding the package from sources, it seems to work.

julia> using SpinePeriods
[ Info: Precompiling SpinePeriods [163584a9-52e1-4a4e-9262-a8c2d5456af4]

julia>

Thank you both @DillonJ and @Tasqu for your assistances.

Regards,

Peng

RowanPeng commented 1 year ago

Hi,

For the fourth line in this step, may I know that I should create a JSON file in advance in the path and then direct it to json_out, or just direct json_out to the output file? Many thanks.

using SpinePeriods
db_url_in = "sqlite:///<path_to_your_input_database>"
db_url_out = "sqlite:///<path_to_your_output_database>"
json_out = "<path_to_your_output_file>.json" 
run_spine_periods(
    db_url_in,
    db_url_out, # replace this with `json_out` to write results to a JSON file
    with_optimizer=HiGHS.Optimizer
)

Regards,

Peng

RowanPeng commented 1 year ago

Hi @Tasqu and @DillonJ ,

Hope you all good. I have two questions, one regarding SpinePeriods example, one regarding error I met.

The first thing is about the test example in the sources. I referred to the parameters settings in SpinePeriod.jl-master/test/Belgium_2017_find database. I was confused about why the deterministic stochastic structure should be related to all nodes and units, which could be seen in the following screenshots.

The second thing is some errors when I simulated in Julia. In the input database, I set unit_duration as minutes which is the same as default, temperal/block_end as 1D, temperal/resolution as 1h, model/roll forward as 1D. And I was trying to find the representative days, where I set representative periods method as representative periods. I set my parameters almost the same as what was set in SpinePeriod.jl-master/test/Belgium_2017_find database, however, there is an error, which is shown as following. I was wondering that you have any ideas or suggestions about what was going on and what I should do.

[ Info: Reading database...
[ Info: Initializing model...
[ Info: Generating SpinePeriods temporal structure...
[ Info: Preprocessing data structure...
ERROR: MethodError: no method matching roll!(::SpineInterface.TimeSlice, ::Dates.Day; update::Bool)

Closest candidates are:
  roll!(::SpineInterface.TimeSlice, ::Union{Dates.CompoundPeriod, Dates.Period}; refresh) got unsupported keyword argument "update"
   @ SpineInterface C:\Users\Jeremy Peng\.julia\packages\SpineInterface\2z5lM\src\api\time_slice.jl:146

Stacktrace:
  [1] kwerr(::NamedTuple{(:update,), Tuple{Bool}}, ::Function, ::SpineInterface.TimeSlice, ::Dates.Day)
    @ Base .\error.jl:165
  [2] _do_roll_temporal_structure!(m::JuMP.Model, rf::Dates.Day, rev::Bool)
    @ SpineOpt C:\Users\Jeremy Peng\.julia\packages\SpineOpt\Flpc7\src\data_structure\temporal_structure.jl:398
  [3] roll_temporal_structure!(m::JuMP.Model, i::Int64; rev::Bool)
    @ SpineOpt C:\Users\Jeremy Peng\.julia\packages\SpineOpt\Flpc7\src\data_structure\temporal_structure.jl:385
  [4] roll_temporal_structure!
    @ C:\Users\Jeremy Peng\.julia\packages\SpineOpt\Flpc7\src\data_structure\temporal_structure.jl:383 [inlined]
  [5] generate_distributions(m::JuMP.Model)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\preprocess_data_structure.jl:147
  [6] preprocess_data_structure
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\preprocess_data_structure.jl:24 [inlined]
  [7] run_spine_periods_selection(url_in::String, out_file::String; with_optimizer::Type, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:21
  [8] run_spine_periods_selection
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:9 [inlined]
  [9] run_spine_periods(url_in::String, out_file::String; with_optimizer::Type, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods.jl:22
 [10] top-level scope
    @ REPL[10]:1

Thank you for your time and help.

Many thanks,

Peng

manuelma commented 1 year ago

The second error is due to SpineOpt being outdated. You can try upgrading SpineOpt to latest, but we should ensure in Project.toml that the right SpineOpt is used. I thought I have done that but maybe I made a mistake.

RowanPeng commented 1 year ago

The second error is due to SpineOpt being outdated. You can try upgrading SpineOpt to latest, but we should ensure in Project.toml that the right SpineOpt is used. I thought I have done that but maybe I made a mistake.

Hi @manuelma . Thank you for your reply. I tried to updated but it showed no changes. I am going to try to uninstall SpineOpt and clone it from sources. Before I do so, I was wondering whether the uninstall process of SpinOpt would influence other packages, like SpinePeriods etc.

Regards,

Peng

manuelma commented 1 year ago

No worries @RowanPeng - I think it should be ok to proceed, uninstalling a package shouldn't mess with other packages. You could take a look at Julia Pkg's manual (it helped me a lot when I was new to julia), here https://pkgdocs.julialang.org/v1/

RowanPeng commented 1 year ago

@manuelma Thank you for your help and suggestions. Really appreciate it.

RowanPeng commented 1 year ago

Hi teams,

Thank you all for the assistances and suggestions to enable SpinePeriods to successfully simulate in my laptop. However, the command has run for more than three days. I was wondering whether you have any ideas about how long it commonly costs to get the results. And also, just want to make sure everything is correct , I attach a screenshot of Julia window in my computer.

I look foward to hearing back from you.

Regards, Peng

RowanPeng commented 1 year ago

Hi,

Because I don't knwo how long it will simulate the results out, I quited. I tried to change the model time to one month time, representative days to 5 days, and the temperal resolution as 12 hours. (I am not quite sure about whether it is okay to setthe parameters as these.) It turned out to be an error in the end. I was wondering if you have any ideas about what this error is about and how to solve this.

[ Info: Reading database...
[ Info: Initializing model...
[ Info: Generating SpinePeriods temporal structure...
[ Info: Preprocessing data structure...

Presolving model
113 rows, 102 cols, 1936 nonzeros
111 rows, 101 cols, 1070 nonzeros

Solving MIP model with:
   111 rows
   101 cols (31 binary, 0 integer, 0 implied int., 70 continuous)
   1070 nonzeros

( 0.0s) Starting symmetry detection
( 0.0s) Found 2 generators

Solving root node LP relaxation

        Nodes      |    B&B Tree     |            Objective Bounds              |  Dynamic Constraints |       Work

     Proc. InQueue |  Leaves   Expl. | BestBound       BestSol              Gap |   Cuts   InLp Confl. | LpIters     Time

         0       0         0   0.00%   0               inf                  inf        0      0      0        64     0.0s
 R       0       0         0   0.00%   2.809573361     169.3548387       98.34%      121      7      0       103     0.0s
 L       0       0         0   0.00%   36.60058515     85.48387097       57.18%     3164    142      0      1389     0.4s
 L       0       0         0   0.00%   36.60058515     75.80645161       51.72%     2683     81      0      4132     0.7s
 T    1042      66       479  24.53%   43.12090564     72.58064516       40.59%     2351    260    123     75891     4.2s

Solving report
  Status            Optimal
  Primal bound      72.5806451613
  Dual bound        72.5806451613
  Solution status   feasible
                    72.5806451613 (objective)
                    8.70414851306e-14 (bound viol.)
                    1.11022302463e-15 (int. viol.)
                    9.94759830064e-14 (row viol.)
  Timing            6.86 (total)
                    0.00 (presolve)
                    0.00 (postsolve)
  Nodes             2058
  LP iterations     131380 (total)
                    18293 (strong br.)
                    7290 (separation)
                    16665 (heuristics)
[ Info: Model solved. Termination status: OPTIMAL.
ERROR: MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer
Stacktrace:
  [1] reduce_empty(op::Base.MappingRF{SpinePeriods.var"#57#61", Base.BottomRF{typeof(min)}}, #unused#::Type{Any})
    @ Base .\reduce.jl:356
  [2] reduce_empty_iter
    @ .\reduce.jl:379 [inlined]
  [3] reduce_empty_iter
    @ .\reduce.jl:378 [inlined]
  [4] foldl_impl
    @ .\reduce.jl:49 [inlined]
  [5] mapfoldl_impl
    @ .\reduce.jl:44 [inlined]
  [6] #mapfoldl#288
    @ .\reduce.jl:170 [inlined]
  [7] mapfoldl
    @ .\reduce.jl:170 [inlined]
  [8] #mapreduce#292
    @ .\reduce.jl:302 [inlined]
  [9] mapreduce
    @ .\reduce.jl:302 [inlined]
 [10] #minimum#303
    @ .\reduce.jl:791 [inlined]
 [11] minimum(a::Base.Generator{Vector{Any}, SpinePeriods.var"#57#61"})
    @ Base .\reduce.jl:791
 [12] postprocess_results!(m::JuMP.Model, db_url::String, out_file::String, window__static_slice::Dict{Any, Any}; alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\postprocess_results.jl:34
 [13] postprocess_results!
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\postprocess_results.jl:20 [inlined]
 [14] run_spine_periods_selection(url_in::String, out_file::String; with_optimizer::Type, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:32
 [15] run_spine_periods_selection
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:9 [inlined]
 [16] run_spine_periods(url_in::String, out_file::String; with_optimizer::Type, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods.jl:22
 [17] top-level scope
    @ REPL[5]:1

Thank you for your time.

Many thanks,

Peng

DillonJ commented 1 year ago

Hi @RowanPeng Can you post a zipped up version of your model? I see that the problem has no integer variables which suggests something is wrong.

The time limit of the optimisation is set to 600 seconds which unfortunately is hard coded. We have an issue to be able to change to solver options in the datastore.

You can try reducing the representative_blocks parameter to try and speed things up. The main factors influencing the size of the problem are:

representatative_periods
representatative_blocks
duration of the model horizon
roll_forward (the smaller the bigger the problem)

I would not recommend choosing a temporal resolution of 12h, your representative periods won't be very representative with such low resolution.

Just play around with these until the model converges in a reasonable time.

You could also change this line in the code to change the otpimisation tolerance and time limit:

https://github.com/spine-tools/SpinePeriods.jl/blob/d1884e785d0217b632388fb1a8e0cb42f2f7e65e/src/run_spine_periods.jl#L13C1-L15

RowanPeng commented 1 year ago

Hi @DillonJ Thank you for your reply and suggestions. This helps me better understand how SpinePeriods works. I've attached the model for you to look over. Thanks for your help. Peng.zip

DillonJ commented 1 year ago

Hi @RowanPeng

The way SpinePeriods works is that it creates a distribution of each timeseries with a number of "blocks" determined by representative_blocks (the default is 40). You have specified a temporal resolution of 24h in your model and a roll_forward of 7 days. That means, each time series only has 7 values. Therefore, an approximation using 40 blocks would be more detail than the actual data and that doesn't work.

The model is working fine, you are just using inputs that don't make any modelling sense. I would reduce the temporal resolution to hourly and play with the other figures. Just be patient, let the model run for the 60 seconds maximum time limit and then look at the solution.

RowanPeng commented 1 year ago

Hi @DillonJ thank you for your reply. I used to set the temporal resolution of 1 hour and a roll-forward of 1 day. But, it runs for too long. So I changed the parameter settings. I will try it again with a reduced maximum time limit. Really appreciate your help.

Additionally, I was wondering if all works fine, the results would write in the output file? Or it will display on Julia window?

DillonJ commented 1 year ago

Hi @RowanPeng , what do you mean by "too long" the solver time limit is set at 600 seconds, which is only 10 minutes. We have models that run for hours - I wouldn't consider 600 seconds as long.

The basic call looks like this: run_spine_periods(url_in, url_out). SpinePeriods will create an output datastore at url_out which is basically a copy of the input datastore with the parameters and representiative periods added

RowanPeng commented 1 year ago

Hi @DillonJ

I followed this command from README , which is

using SpinePeriods
db_url_in = "sqlite:///<path_to_your_input_database>"
db_url_out = "sqlite:///<path_to_your_output_database>"
json_out = "<path_to_your_output_file>.json" 
run_spine_periods(
    db_url_in,
    db_url_out, # replace this with `json_out` to write results to a JSON file
    with_optimizer=HiGHS.Optimizer
)

The simulation kept running for four days and still didn't get the results. I am confused about whether I used the correct command to simulate the SpinePeriods. This is what I did in the Julia window.

julia> using SpinePeriods, HiGHS

julia> db_url_in = "sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/Generator_databasev2023_06_02.sqlite"
"sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/Generator_databasev2023_06_02.sqlite"

julia> db_url_out = "sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output/OUTPUT.sqlite"
"sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output/OUTPUT.sqlite"

julia> json_out = "C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output.json"
"C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output.json"

julia> run_spine_periods(
           db_url_in,
           db_url_out, # replace this with `json_out` to write results to a JSON file
           with_optimizer=HiGHS.Optimizer
       )
[ Info: Reading database...
[ Info: Initializing model...
[ Info: Generating SpinePeriods temporal structure...
[ Info: Preprocessing data structure...

Presolving model
447 rows, 770 cols, 21090 nonzeros
445 rows, 769 cols, 14214 nonzeros

Solving MIP model with:
   445 rows
   769 cols (365 binary, 0 integer, 0 implied int., 404 continuous)
   14214 nonzeros

( 0.0s) Starting symmetry detection
( 0.1s) No symmetry present

Solving root node LP relaxation

        Nodes      |    B&B Tree     |            Objective Bounds              |  Dynamic Constraints |       Work

     Proc. InQueue |  Leaves   Expl. | BestBound       BestSol              Gap |   Cuts   InLp Confl. | LpIters     Time

         0       0         0   0.00%   0               inf                  inf        0      0      0       419     0.1s
 R       0       0         0   0.00%   0               177.8500761      100.00%       83      7      0       963     0.3s
 L       0       0         0   0.00%   0               13.55339608      100.00%      376     21      0      1911     2.0s
 L       0       0         0   0.00%   0               11.0623753       100.00%      376     11      0      5477     3.7s
 B       0       0         0   0.00%   0               6.482573397      100.00%      376     11      0      8425     4.7s
 B       0       0         0   0.00%   0               5.353973215      100.00%      384     11      0      8425     5.1

DillonJ commented 1 year ago

Hi @RowanPeng,

So four days sounds excessive - HiGHS must not be obeying the time limit for some reason - perhaps it's because you have specified the solver in the function call. Can you try this instead?

using SpinePeriods
db_url_in = "sqlite:///<path_to_your_input_database>"
db_url_out = "sqlite:///<path_to_your_output_database>"
json_out = "<path_to_your_output_file>.json" 
run_spine_periods(
    db_url_in,
    db_url_out, # replace this with `json_out` to write results to a JSON file
    with_optimizer=optimizer_with_attributes(
        HiGHS.Optimizer, "output_flag" => true, "mip_rel_gap" => 0.01, "time_limit" => 600.0        
    )
)

The problem was that in specifying the solver in the call to run_spine_periods, it was overriding the default which had a time limit specified. In the above, we have added the HiGHS solver options mip_rel_gap (the optimality tolerance) and time_limit which is the time limit in seconds. We should fix the readme to include these by default. You should now get a solution in 600 seconds... the longer you make it, the better the solution could be, but 10 minutes should be fine to get an idea if it's working.

Then your results will be in the datastore specified at db_url_out which in your case will be "sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output/OUTPUT.sqlite". Spine periods will create a copy of your datastore at this path with the period data added (specifically, the representative period mapping and window weights).

Note that you specify either db_url_out or json_out not both. You will see that in your call to run_spine_opt you only use db_url_out so it's not necessary to specify json_out.

I hope that helps

RowanPeng commented 1 year ago

Hi @DillonJ Thank you for your reply.

I followed your instructions and removed theusing HiGHS. Then, the error reappears, saying that HiGHS not defines.

julia> using SpinePeriods
[ Info: Precompiling SpinePeriods [163584a9-52e1-4a4e-9262-a8c2d5456af4]

julia> db_url_in = "sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/Generator_databasev2023_06_02.sqlite"
"sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/Generator_databasev2023_06_02.sqlite"

julia> db_url_out = "sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output/OUTPUT.sqlite"
"sqlite:///C:/Users/Jeremy Peng/Desktop/Test_GB_model_testing_representative_2/Test_GB_model_testing_representative/Test_GB_model_testing/.spinetoolbox/items/output/OUTPUT.sqlite"

julia> run_spine_periods(
           db_url_in,
           db_url_out, # replace this with `json_out` to write results to a JSON file
           with_optimizer=optimizer_with_attributes(
               HiGHS.Optimizer, "output_flag" => true, "mip_rel_gap" => 0.01, "time_limit" => 600.0
           )
       )
ERROR: UndefVarError: `HiGHS` not defined
Stacktrace:
 [1] top-level scope
   @ REPL[4]:1

DillonJ commented 1 year ago

Why did you remove using HiGHS? The solution is to put it back in. What I meant was that in the instructions you followed, the "with_optimizer" keyword argument was specified with a value of optimizer_with_attributes(HiGHS.Optimizer). Here, no solver options are specified. This overrode the default in the code which does specify the options - so that's why the time limit wasn't respected.

In short, just put using HiGHS back in and it should work

RowanPeng commented 1 year ago

Hi @DillonJ Thanks for pointing out the fault.

However, there is a new error appearing, which said that mip_rel_gap is not defined. I reinstalled JuMP and HiGHS packages. I searched for it, but couldn't find a solution. Do you have any ideas about that?

ERROR:   getOptionIndex: Option "mip_rel_gap" is unknown
ERROR: MathOptInterface.UnsupportedAttribute{MathOptInterface.RawParameter}: Attribute MathOptInterface.RawParameter("mip_rel_gap") is not supported by the model.
Stacktrace:
  [1] set(model::HiGHS.Optimizer, param::MathOptInterface.RawParameter, value::Float64)
    @ HiGHS C:\Users\Jeremy Peng\.julia\packages\HiGHS\YP79q\src\MOI_wrapper.jl:503
  [2] _instantiate_and_check(optimizer_constructor::MathOptInterface.OptimizerWithAttributes)
    @ MathOptInterface C:\Users\Jeremy Peng\.julia\packages\MathOptInterface\YDdD3\src\instantiate.jl:88
  [3] instantiate(optimizer_constructor::MathOptInterface.OptimizerWithAttributes; with_bridge_type::Type{Float64}, with_names::Bool)
    @ MathOptInterface C:\Users\Jeremy Peng\.julia\packages\MathOptInterface\YDdD3\src\instantiate.jl:120
  [4] set_optimizer(model::Model, optimizer_constructor::MathOptInterface.OptimizerWithAttributes; bridge_constraints::Bool)
    @ JuMP C:\Users\Jeremy Peng\.julia\packages\JuMP\klrjG\src\optimizer_interface.jl:109
  [5] set_optimizer
    @ C:\Users\Jeremy Peng\.julia\packages\JuMP\klrjG\src\optimizer_interface.jl:99 [inlined]
  [6] #Model#23
    @ C:\Users\Jeremy Peng\.julia\packages\JuMP\klrjG\src\JuMP.jl:287 [inlined]
  [7] Model
    @ C:\Users\Jeremy Peng\.julia\packages\JuMP\klrjG\src\JuMP.jl:285 [inlined]
  [8] run_spine_periods_selection(url_in::String, out_file::String; with_optimizer::MathOptInterface.OptimizerWithAttributes, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:16
  [9] run_spine_periods_selection
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:9 [inlined]
 [10] run_spine_periods(url_in::String, out_file::String; with_optimizer::MathOptInterface.OptimizerWithAttributes, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods.jl:22
 [11] top-level scope
    @ REPL[6]:1

DillonJ commented 1 year ago

I had this before here: https://github.com/spine-tools/SpinePeriods.jl/issues/11#issuecomment-1680193961

Can you post the output of

using Pkg
Pkg.status()

RowanPeng commented 1 year ago

Status `C:\Users\Jeremy Peng\.julia\environments\v1.9\Project.toml`
⌃ [a076750e] CPLEX v0.7.8
⌃ [9961bab8] Cbc v0.8.1
⌃ [e2554f3b] Clp v0.8.4
  [8f4d0f93] Conda v1.9.1
⌃ [87dc4568] HiGHS v0.2.3
  [682c06a0] JSON v0.21.4
⌃ [4076af6c] JuMP v0.21.10
⌃ [2ddba703] Juniper v0.7.0
  [438e738f] PyCall v1.96.1
  [0cda1612] SpineInterface v0.11.3 `C:\Users\Jeremy Peng\SpineInterface.jl`
  [0d8fc150] SpineOpt v0.6.13 `C:\Users\Jeremy Peng\SpineOpt.jl`
  [163584a9] SpinePeriods v0.1.1 `C:\Users\Jeremy Peng\SpinePeriods.jl`
⌃ [5c2747f8] URIs v1.4.2
  [3a884ed6] UnPack v1.0.2
Info Packages marked with ⌃ have new versions available and may be upgradable.

Yes, I saw that. And I also tried to update and reinstalled, but nothing changed.

DillonJ commented 1 year ago

I think it might be CLP, CBC limiting your version of Jump which in turn is limiting your version of HIGHs. I would remove Clp and Cbc then upgrade Jump then upgrade HiGHS. So:

Pkg.rm("Clp")
Pkg.rm("Cbc")
Pkg.update("JuMP")
Pkg.update("HiGHS")

Edit: alternatively, updating Clp and Cbc might do it also, if you have any problems after removing them Edit2: also update CPLEX (before updating JuMP and HiGHS)

RowanPeng commented 1 year ago

Hi @DillonJ

I reinstalled JuMP locally, and it works. But I think it was due to the time limit. It didn't find the optimal solution. Should I change the time limit to a larger value?

[ Info: Model solved. Termination status: TIME_LIMIT.
ERROR: MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer
Stacktrace:
  [1] reduce_empty(op::Base.MappingRF{SpinePeriods.var"#57#61", Base.BottomRF{typeof(min)}}, #unused#::Type{Any})
    @ Base .\reduce.jl:356
  [2] reduce_empty_iter
    @ .\reduce.jl:379 [inlined]
  [3] reduce_empty_iter
    @ .\reduce.jl:378 [inlined]
  [4] foldl_impl
    @ .\reduce.jl:49 [inlined]
  [5] mapfoldl_impl
    @ .\reduce.jl:44 [inlined]
  [6] #mapfoldl#288
    @ .\reduce.jl:170 [inlined]
  [7] mapfoldl
    @ .\reduce.jl:170 [inlined]
  [8] #mapreduce#292
    @ .\reduce.jl:302 [inlined]
  [9] mapreduce
    @ .\reduce.jl:302 [inlined]
 [10] #minimum#303
    @ .\reduce.jl:791 [inlined]
 [11] minimum(a::Base.Generator{Vector{Any}, SpinePeriods.var"#57#61"})
    @ Base .\reduce.jl:791
 [12] postprocess_results!(m::Model, db_url::String, out_file::String, window__static_slice::Dict{Any, Any}; alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\postprocess_results.jl:34
 [13] postprocess_results!
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\postprocess_results.jl:20 [inlined]
 [14] run_spine_periods_selection(url_in::String, out_file::String; with_optimizer::MathOptInterface.OptimizerWithAttributes, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:32
 [15] run_spine_periods_selection
    @ C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods_selection.jl:9 [inlined]
 [16] run_spine_periods(url_in::String, out_file::String; with_optimizer::MathOptInterface.OptimizerWithAttributes, alternative::String)
    @ SpinePeriods C:\Users\Jeremy Peng\SpinePeriods.jl\src\run_spine_periods.jl:22
 [17] top-level scope
    @ REPL[10]:1

DillonJ commented 1 year ago

What settings are you using for temporal resolution, representative_blocks and periods? Did you reset to hourly resolution?

RowanPeng commented 1 year ago

Hi @DillonJ

I set the temporal resolution as hourly, representative_blocks as default (40), periods as 14, and roll_forwad as 1 day.

DillonJ commented 1 year ago

The problem is that you have specified neither node__temporal_block for the electricity node, nor the model__default_temporal_block relationship. The easiest is to set the model__default_temporal_block relationship and that will apply to all nodes in the model unless you specify a node__temporal_block which would override the default.

Also, it looks like the only timeseries that you are trying to represent is the demand of the electricity node. I don't see any renewable generation in the model. Normally you would be trying to find periods that best represent the possible combinations of demand and renewable generation. Right now, you will be finding periods that best represent the distribution of the electricity demand only.

RowanPeng commented 1 year ago

Hi @DillonJ Thank you for looking at that.

If I set model_default_temporal_block to apply to all nodes, I was wondering whether it would override to set cyclic_condition parameters for the storage nodes.

DillonJ commented 1 year ago

You override the default by specifying individual node__temporal_block relationships so it’s not a problem Sent from my iPhoneOn 16 Aug 2023, at 12:28, RowanPeng @.***> wrote: Hi @DillonJ Thank you for looking at that. If I set model_default_temporal_block to apply to all nodes, I was wondering whether it would override to set cyclic_condition parameters for the storage nodes.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

RowanPeng commented 1 year ago

@DillonJ Thanks for your help. It makes sense to me. I could get the results now. I will try to find the optimal parameter settings to get the best results.

spine-tools / SpinePeriods.jl

Representative periods ordering and mapping settings #11