search

EnergyInnovation / eps-us

Energy Policy Simulator - United States
GNU General Public License v3.0
22 stars 7 forks source link

Bug in policy schedule implementation for TDM policies #114

Closed robbieorvis closed 3 years ago

robbieorvis commented 3 years ago

We discovered today that setting TDM policies was causing the model to crash, which flagged a bug in Vensim. In the new policy schedule implementation methodology for the revised TDM levers, the schedule calculation is not correctly subscripted by policy element, and so all elements relate back to the LDV passenger policy schedule setting.

jrissman commented 3 years ago

I think you might be misinterpreting the syntax of the GET ELM MAP() function. It's supposed to contain a reference to the first element of the subscript from which you want to start mapping. Look at the equation inside another subscripted policy lever's "This Year" version to see other examples, like "Boolean Rebate Program for Efficient Components This Year."

I'll run some tests to see if it's working or not.

robbieorvis commented 3 years ago

Ah okay – yes in that case I was misinterpreting. However, there is still a crash (we are using the eps-nevada model on the develop branch; you can replicate the issue by just turning on a freight HDV TDM policy (I used a value of 0.22).

-R

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 1:56 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

I think you might be misinterpreting the syntax of the GET ELM MAP() function. It's supposed to contain a reference to the first element of the subscript from which you want to start mapping. Look at the equation inside another subscripted policy lever's "This Year" version to see other examples, like "Boolean Rebate Program for Efficient Components This Year."

I'll run some tests to see if it's working or not.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729885631, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SJ3PMO4LCMXIRVUXJTSQQKCFANCNFSM4T2LCHDA.

jrissman commented 3 years ago

In my testing of the U.S. EPS, this policy obeys the schedules for each subscript element normally. I don't encounter an issue where it is using the first element's schedule for all elements. (The easiest way that can happen if you forget to set the Policy Implementation Schedule selector to the custom schedule you made, because the default schedule contains the same setting for every element of every policy.)

I'll look into the crash using EPS Nevada because I can't recreate any problematic behavior in the U.S. EPS.

jrissman commented 3 years ago

Does eps-nevada have a repo? If so, maybe I was never invited as a collaborator. I don't see it in the EI account's repo list.

robbieorvis commented 3 years ago

It’s a private repo on the EI GitHub account – you should see it there

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:10 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

Does eps-nevada have a repo? If so, maybe I was never invited as a collaborator. I don't see it in the EI account's repo list.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729893413, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SORRUIUNRJKN7WNBL3SQQLZFANCNFSM4T2LCHDA.

jrissman commented 3 years ago

I do not see it there. I can log in as EI GitHub admin and make sure I was added as a collaborator, which is necessary to see private repos when logged in as myself.

robbieorvis commented 3 years ago

Oh yes, I thought that’s what you were planning to do.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:16 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

I do not see it there. I can log in as EI GitHub admin and make sure I was added as a collaborator, which is necessary to see private repos when logged in as myself.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729896393, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SLQPU5NEJIO2UWZFV3SQQMN5ANCNFSM4T2LCHDA.

robbieorvis commented 3 years ago

I think I see what may be the source of the problem, though not the cause. Freight aircraft have 0 cargo distance transported demand in BAU and zero vehicles, which I think is why we are seeing an error. But I’m not quite sure how this plays out with the new structure.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:16 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

I do not see it there. I can log in as EI GitHub admin and make sure I was added as a collaborator, which is necessary to see private repos when logged in as myself.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729896393, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SLQPU5NEJIO2UWZFV3SQQMN5ANCNFSM4T2LCHDA.

jrissman commented 3 years ago

Those freight figures shouldn't be a problem, if they are accurate for Nevada.

Right now, I'm investigating whether the Nevada multipliers in the I/O model are too high for non-road vehicle manufacturers. When you shift from freight HDVs, it goes to freight rail. That boosts the non-road vehicle manufacturers' ISIC code. If that ISIC code has unreasonable multipliers, any but the smallest boost to that ISIC code can cause a crash.

robbieorvis commented 3 years ago

One extremely odd thing is that when I toggle on the freight HDVs lever to any policy setting, and I click on Table, it shows zeroes. I have no idea what that’s about – never seen it before, unless it’s doing that because of the calculation error.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:38 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

Wait, I may have been looking at the wrong branch. Hold on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729908616, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SKYUHPQCMP6OIHTYMTSQQPCFANCNFSM4T2LCHDA.

jrissman commented 3 years ago

Incidentally, you have trans/RTMF set up to send 10% of cargo from freight HDVs freight ships, and that would be quite a feat in Nevada!

robbieorvis commented 3 years ago

Well – it’s seeming to work with smaller policy values than 0.2 (I tried 0.01 and no error)

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:38 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

Wait, I may have been looking at the wrong branch. Hold on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729908616, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SKYUHPQCMP6OIHTYMTSQQPCFANCNFSM4T2LCHDA.

jrissman commented 3 years ago

One extremely odd thing is that when I toggle on the freight HDVs lever to any policy setting, and I click on Table, it shows zeroes. I have no idea what that’s about – never seen it before, unless it’s doing that because of the calculation error.

If Vensim crashes, it doesn't necessarily report correct values for any variable, even in timesteps prior to the crash. This is one way Vensim makes it hard to debug crash issues.

robbieorvis commented 3 years ago

Okay, I think this may eventually come down to a data issue. When I turn on the policy setting for HDV freight to 0.05, I can see a huge drop in cargo distance transported in later years as a product of some feedback effect. I’m trying to trace where this is stemming from, but my guess is it’s an input data issue.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:48 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

Those freight figures shouldn't be a problem, if they are accurate for Nevada.

Right now, I'm investigating whether the Nevada multipliers in the I/O model are too high for non-road vehicle manufacturers. When you shift from freight HDVs, it goes to freight rail. That boosts the non-road vehicle manufacturers' ISIC code. If that ISIC code has unreasonable multipliers, any but the smallest boost to that ISIC code can cause a crash.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729913656, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SMI2U6YFA2CJ5H7DVDSQQQF3ANCNFSM4T2LCHDA.

jrissman commented 3 years ago

It looks like io-model/DCSoCbIC Domestic Content Share of Consumption by ISIC Code is probably far too high. I cannot believe that such overwhelming percentages of things purchased in Nevada were made by Nevada businesses. I thought RMI had a script-based methodology or data source to downscale DCSoCbIC to the state level. (A procedure I once suggested to Jun to produce DLIM for state-level models relies on having DCSoCbIC data, so if you don't have reasonable DCSoCbIC data, maybe DLIM isn't reasonable either, though I haven't checked DLIM directly yet.)

robbieorvis commented 3 years ago

Turning off the macroeconomic effects eliminated the problem, and I’m noticing massive job and GDP impacts, far too big to be from this little policy. We need to double check the input data.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 2:53 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

One extremely odd thing is that when I toggle on the freight HDVs lever to any policy setting, and I click on Table, it shows zeroes. I have no idea what that’s about – never seen it before, unless it’s doing that because of the calculation error.

If Vensim crashes, it doesn't necessarily report correct values for any variable, even in timesteps prior to the crash. This is one way Vensim makes it hard to debug crash issues.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729916373, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SK5IB44VUNONKKH4CTSQQQZ5ANCNFSM4T2LCHDA.

robbieorvis commented 3 years ago

They do, and we have flagged this multiple times for them. Let us circle back to them on this.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 3:00 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

It looks like io-model/DCSoCbIC Domestic Content Share of Consumption by ISIC Code is probably far too high. I cannot believe that such overwhelming percentages of things purchased in Nevada were made by Nevada businesses. I thought RMI had a script-based methodology or data source to downscale DCSoCbIC to the state level. (A procedure I once suggested to Jun to produce DLIM for state-level models relies on having DCSoCbIC data, so if you don't have reasonable DCSoCbIC data, maybe DLIM isn't reasonable either, though I haven't checked DLIM directly yet.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729919663, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SMQK5ULBAJJ575TRF3SQQRSJANCNFSM4T2LCHDA.

robbieorvis commented 3 years ago

I see – the .xlsx file is correct but it didn’t get exported to .csv. You can close this issue.

Robbie Orvis Director of Energy Policy Design Phone: 415-799-2171 98 Battery Street, Suite 202 San Francisco, CA 94111 www.energyinnovation.orghttp://www.energyinnovation.org/ [cid:image001.jpg@01D0D699.20A24470]

Check out our new book, Designing Climate Solutions: A Policy Guide for Low-Carbon Energyhttps://www.amazon.com/Designing-Climate-Solutions-Policy-Low-Carbon/dp/1610919564 Available wherever books are sold

[Policy Design book cover]

From: Jeff Rissman notifications@github.com Sent: Wednesday, November 18, 2020 3:00 PM To: Energy-Innovation/eps-us eps-us@noreply.github.com Cc: Robbie Orvis robbie@energyinnovation.org; Author author@noreply.github.com Subject: Re: [Energy-Innovation/eps-us] Bug in policy schedule implementation for TDM policies (#114)

It looks like io-model/DCSoCbIC Domestic Content Share of Consumption by ISIC Code is probably far too high. I cannot believe that such overwhelming percentages of things purchased in Nevada were made by Nevada businesses. I thought RMI had a script-based methodology or data source to downscale DCSoCbIC to the state level. (A procedure I once suggested to Jun to produce DLIM for state-level models relies on having DCSoCbIC data, so if you don't have reasonable DCSoCbIC data, maybe DLIM isn't reasonable either, though I haven't checked DLIM directly yet.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Energy-Innovation/eps-us/issues/114#issuecomment-729919663, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5N6SMQK5ULBAJJ575TRF3SQQRSJANCNFSM4T2LCHDA.

jrissman commented 3 years ago

As a good starting point, why don't you try using the "Input Data Debugging Helper" I made for EPS Virginia. Using that tool should probably be a standard part of calibration now that we have an IO model. That will check the reasonableness of DLIM and the three within-industry intensity multipliers. (It does not check DCSoCbIC, but maybe it should.)

jrissman commented 3 years ago

Okay, I'll close it.