search

FSEC-Photovoltaics / pvrpm-lcoe

PV Simulation using SAM and Python
BSD 3-Clause "New" or "Revised" License
8 stars 2 forks source link

BUG (maybe): Can't run the simulation #49

Open SarthakJariwala opened 2 years ago

SarthakJariwala commented 2 years ago

While trying to run pvrpm with a test case and example config file, I am getting the following error - Illegal instruction while running pvrpm (see below for more details).

Error

$ pvrpm run --case test-case-comm example_config.yml --trace

2022-02-22 16:20:21,419--WARNING: There is a non-zero value in the fixed annual O&M costs input. These will be overwritten with the new values.
2022-02-22 16:20:21,419--WARNING: Degradation is set by the PVRPM script, you have entered a non-zero degradation to the degradation input. This script will set the degradation input to zero.
[1]    49994 illegal hardware instruction  pvrpm run --case test-case-comm example_config.yml --trace

Details

I am running everything on 'macOS-10.16-x86_64-i386-64bit'

Installation

I am using conda to create a new environment and install pvrpm as follows:

conda create -n pvrpm "python>=3.8"
conda activate pvrpm
python -m pip install git+https://github.com/FSEC-Photovoltaics/pvrpm-lcoe/@v1.7.1

SAM

I am using the "Residential Owner" model under "Detailed PV Model" and exporting the PySAM JSON to a folder.

Config file

I downloaded the example config file and changed the results folder.

Complete specs of packages

# packages in environment at /Users/sjariwala/miniconda3/envs/pvrpm:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h0d85af4_4    conda-forge
ca-certificates           2021.10.8            h033912b_0    conda-forge
certifi                   2021.10.8        py38hecd8cb5_2
click                     8.0.4                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
fonttools                 4.29.1                   pypi_0    pypi
kiwisolver                1.3.2                    pypi_0    pypi
libcxx                    12.0.1               habf9029_1    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libzlib                   1.2.11            h9173be1_1013    conda-forge
matplotlib                3.5.1                    pypi_0    pypi
ncurses                   6.3                  he49afe7_0    conda-forge
nrel-pysam                3.0.0                    pypi_0    pypi
nrel-pysam-stubs          3.0.0                    pypi_0    pypi
numpy                     1.22.2                   pypi_0    pypi
openssl                   3.0.0                h0d85af4_2    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    1.4.1                    pypi_0    pypi
pillow                    9.0.1                    pypi_0    pypi
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
pvrpm                     1.7.1                    pypi_0    pypi
pyparsing                 3.0.7                    pypi_0    pypi
python                    3.8.12          h43ca1e7_3_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2021.3                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.1                  h05e3726_0    conda-forge
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py38hecd8cb5_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.37.0               h23a322b_0    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
tqdm                      4.62.3                   pypi_0    pypi
tzdata                    2021e                he74cb21_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
zlib                      1.2.11            h9173be1_1013    conda-forge
SarthakJariwala commented 2 years ago

Also, using the verify command results in the same error :

$ pvrpm verify --case test-case-comm example_config.yml

2022-02-22 16:43:40,795--WARNING: There is a non-zero value in the fixed annual O&M costs input. These will be overwritten with the new values.
2022-02-22 16:43:40,795--WARNING: Degradation is set by the PVRPM script, you have entered a non-zero degradation to the degradation input. This script will set the degradation input to zero.
[1]    50513 illegal hardware instruction  pvrpm verify --case test-case-comm example_config.yml
SarthakJariwala commented 2 years ago

Maybe this helps in your debugging - I couldn't run the tests as well using pytest (see output below)

$ pytest
=========================================================== test session starts ============================================================
platform darwin -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/sjariwala/Documents/JOSS_review/pvrpm-lcoe
collected 5 items

tests/integration/test_simulation.py Fatal Python error: Illegal instruction

Current thread 0x00000001189c8600 (most recent call first):
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pvrpm/core/case.py", line 763 in value
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pvrpm/core/case.py", line 540 in __verify_case
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pvrpm/core/case.py", line 70 in __init__
  File "/Users/sjariwala/Documents/JOSS_review/pvrpm-lcoe/tests/integration/test_simulation.py", line 17 in test_simulation
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/python.py", line 1718 in runtest
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 261 in <lambda>
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 340 in from_call
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/main.py", line 322 in _main
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/Users/sjariwala/miniconda3/envs/pvrpm/lib/python3.8/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/Users/sjariwala/miniconda3/envs/pvrpm/bin/pytest", line 8 in <module>
[1]    51082 illegal hardware instruction  pytest
brandons209 commented 2 years ago

Some quick googling suggests this is a macOS specific issue with running Python on it, more specifically to your processor. My guess is that one of the packages PVRPM uses is not supported for your CPU architecture, even though I think you are using an Intel x86_64 architecture processor, although I am not familiar with mac products. If you have the M1 chip that may be the issue. Unfortunately, I do not have easy access to a macOS device for testing. The error when running verify suggests Python itself is segfaulting with that error, which various articles on the Google search seem to confirm. I may be able to get a hold of a macOS device for testing but I am not sure right now, may have to try to setup a VM, but a VM environment may not produce the specific issue you are having.

My best guess is that is probably pandas or numpy causing the issue, so maybe finding and installing mac specific versions for those packages or researching issues along with those packages on macOS? I'll try to do some digging and post anything relevant I can find here. It may be that PVRPM just won't work on some macOS computers.

SarthakJariwala commented 2 years ago

I am running it on a Mac with Intel chip.

I don't think it is related to numpy or pandas. I just tried running some NumPy and pandas-related computation in the pvrpm environment I created, and everything works as it should have. I have also been using NumPy and pandas in other environments without any issues.

One thing I would suggest is to add macOS to your test workflow using github actions and see if the tests pass/run. It might be faster than getting hold of a mac. (Along those lines, I see you are already testing on ubuntu but I would also recommend adding macOS and windows to your test workflow)

brandons209 commented 2 years ago

Alright, I didn't realize actions can test on multiple OSes. I'll go ahead and add them and see what happens. If its not those packages then something is causing Python itself to segfault, which would be hard to diagnose.

brandons209 commented 2 years ago

Alright, so the action fails on macOS. It has to be one of the packages causing the issue. Could you do me a favor?

Can you import each library PVRPM uses one at a time and see if any of them cause the crash? Just run each code snippet one at a time in a interpreter and see if any cause a crash.

import PySAM.PySSC as pssc
ssc = pssc.PySSC()

Loading all PySAM packages:

import pkgutil
import importlib
def load_pysam_modules():
    global pysam
    import PySAM as pysam

    for loader, module_name, is_pkg in pkgutil.walk_packages(pysam.__path__):
        try:
            print(f"Loading: {module_name}")
            importlib.import_module(f"{pysam.__name__}.{module_name}")
        except:
            pass

load_pysam_modules()
import scipy
import scipy.stats as stats
from scipy.special import gamma, gammaln

import click

import yaml

import matplotlib.pyplot as plt

from dateutil.relativedelta import relativedelta

from tqdm import tqdm

import numpy as np
import pandas as pd
brandons209 commented 2 years ago

Hopefully one of those will cause the crash and I can see if there is a workaround for that package.

SarthakJariwala commented 2 years ago

I changed the except: pass to print the Exception in PySAM snippet and got the following Exceptions (complete output is underneath that) -

All other snippets worked.

dynamic module does not define module export function (PyInit_libSAM_api)
dynamic module does not define module export function (PyInit_libssc)
No module named 'requests'

Complete output from PySAM snippet:

Loading : AdjustmentFactors
Loading : Annualoutput
Loading : Battery
Loading : BatteryStateful
Loading : BatteryTools
Loading : Battwatts
Loading : Belpe
Loading : Biomass
Loading : Cashloan
Loading : CbConstructionFinancing
Loading : CbEmpiricalHceHeatLoss
Loading : CbMsptSystemCosts
Loading : Communitysolar
Loading : DsgFluxPreprocess
Loading : Equpartflip
Loading : EtesElectricResistance
Loading : Fuelcell
Loading : GenericSystem
Loading : Geothermal
Loading : GeothermalCosts
Loading : Grid
Loading : Hcpv
Loading : HostDeveloper
Loading : Iec61853interp
Loading : Iec61853par
Loading : InvCecCg
Loading : IphToLcoefcr
Loading : Ippppa
Loading : Irradproc
Loading : IsccDesignPoint
Loading : Layoutarea
Loading : Lcoefcr
Loading : Levpartflip
Loading : LinearFresnelDsgIph
Loading : Merchantplant
Loading : MhkCosts
Loading : MhkTidal
Loading : MhkWave
Loading : Poacalib
Loading : Pv6parmod
Loading : PvGetShadeLossMpp
Loading : Pvsamv1
Loading : Pvsandiainv
Loading : Pvwattsv1
Loading : Pvwattsv11ts
Loading : Pvwattsv1Poa
Loading : Pvwattsv5
Loading : Pvwattsv51ts
Loading : Pvwattsv7
Loading : Pvwattsv8
Loading : PySSC
Loading : ResourceTools
No module named 'requests'
Loading : Saleleaseback
Loading : Sco2AirCooler
Loading : Sco2CompCurves
Loading : Sco2CspSystem
Loading : Sco2CspUdPcTables
Loading : Singlediode
Loading : Singlediodeparams
Loading : Singleowner
Loading : SixParsolve
Loading : Snowmodel
Loading : Solarpilot
Loading : Swh
Loading : TcsMSLF
Loading : TcsgenericSolar
Loading : TcslinearFresnel
Loading : TcsmoltenSalt
Loading : TcstroughEmpirical
Loading : TcstroughPhysical
Loading : TestUdPowerCycle
Loading : Thermalrate
Loading : Thirdpartyownership
Loading : Timeseq
Loading : TroughPhysical
Loading : TroughPhysicalCspSolver
Loading : TroughPhysicalProcessHeat
Loading : UiTesCalcs
Loading : UiUdpcChecks
Loading : UserHtfComparison
Loading : Utilityrate
Loading : Utilityrate2
Loading : Utilityrate3
Loading : Utilityrate4
Loading : Utilityrate5
Loading : WaveFileReader
Loading : Wfcheck
Loading : Wfcsvconv
Loading : Wfreader
Loading : WindFileReader
Loading : WindObos
Loading : Windbos
Loading : Windcsm
Loading : Windpower
Loading : libSAM_api
dynamic module does not define module export function (PyInit_libSAM_api)
Loading : libssc
dynamic module does not define module export function (PyInit_libssc)
Loading : version
Loading : version_differ
No module named 'requests'
brandons209 commented 2 years ago

Those are fine, but every snippet worked? That's really odd then...

If verify causes a crash then we can try narrowing it down a bit. Its not with the SAM simulation or PVRPM simulation.

Its crashing somewhere in creating the SAM case in PVRPM.

If you can clone the repo to your local computer, then cd into the repo and do pip install -e .

Then go to pvrpm > core > case.py and add in print statements after every line in the init function to try to narrow down where this crash occurs.

Something like:

def __init__(self, sam_json_dir: str, config: str, num_realizations: int = 0, results_folder: str = None):
        """"""
        print("enter init")
        self.ssc = pssc.PySSC()
        print("test 1")
        self.config = self.__load_config(config, type="yaml")
        print("test 2")
        self.sam_json_dir = sam_json_dir
        self.daily_tracker_coeffs = None
        self.modules = self.__load_modules()
        print("test 3")

        # will be calculated after base case simulation
        self.daylight_hours = None
        self.annual_daylight_hours = None
        self.base_lcoe = None
        self.base_npv = None
        self.base_ac_energy = None
        self.base_annual_energy = None
        self.base_dc_energy = None

        self.base_load = None
        self.base_tax_cash_flow = None
        self.base_losses = {}

        if not (self.modules and self.config):
            raise CaseError("There are errors in the configuration files, see logs.")

        # override results folder and number of realizations
        print("test 4")
        if num_realizations >= 2:
            self.config[ck.NUM_REALIZATION] = num_realizations
        if results_folder is not None:
            self.config[ck.RESULTS_FOLDER] = results_folder

        # lookup table for module order:
        self.module_orders = [
            ["Pvsamv1", "Grid", "Utilityrate5", "Cashloan"],
            ["Pvsamv1", "Grid", "Utilityrate5", "Merchantplant"],
            ["Pvsamv1", "Grid", "Utilityrate5", "Levpartflip"],
            ["Pvsamv1", "Grid", "Utilityrate5", "Equpartflip"],
            ["Belpe", "Pvsamv1", "Grid", "Utilityrate5", "Cashloan"],
            ["Pvsamv1", "Grid", "Utilityrate5", "Saleleaseback"],
            ["Pvsamv1", "Grid", "Utilityrate5", "Singleowner"],
            ["Pvsamv1", "Grid", "Utilityrate5", "HostDeveloper"],
            ["Belpe", "Pvsamv1", "Grid", "Utilityrate5", "Thirdpartyownership"],
        ]

        # lookup table for models that pvrpm cannot use
        self.bad_module_orders = [
            ["Pvsamv1", "Grid"],
            ["Pvsamv1", "Grid", "Lcoefcr"],
        ]
        print("test 5")
        self.__verify_case()
        print("test 6")
        self.__verify_config()
        print("test 7")

Then run the pvrpm verify command and see where it crashes. Rince and repeat narrowing it down to the line that causes it.

Just try and narrow down where this is happening. You can also try using the debugger but I haven't had much experience with the Python debugger, if you do and want to try that please go ahead.

brandons209 commented 2 years ago

If you don't have the time or unable I can try to do it with the github action as well, just let me know.

SarthakJariwala commented 2 years ago

I won't be able to give it a try until the weekend. Please go ahead and try it with GitHub Actions.

brandons209 commented 2 years ago

I have no clue how to even tackle this. It seems adding in logging statements around the lines that cause this issue seems to prevent the issue from occuring. It happened in a few different spots but I think its occurring when trying to get a value from the PySAM modules. It makes no sense that adding in a logging statement before and after those lines makes them run fine.

brandons209 commented 2 years ago

Could you try running the tests for PySAM itself? I suspect that might help reveal the issue. The repo is here: https://github.com/NREL/pysam

I found an issue on pysam's github that relates to a segfault running on macOS 10.16. I think maybe those who use PySAM don't use macOS and that it just doesn't work on that OS.

SarthakJariwala commented 2 years ago

It makes no sense that adding in a logging statement before and after those lines makes them run fine.

Yeah, that's very odd! Can you push those changes to main or another branch, and I can try it out whenever I get some time?

Could you try running the tests for PySAM itself?

I ran PySAM tests on my mac, and they passed. (It could be that their tests aren't comprehensive; I couldn't find any test coverage reports)

I also tried re-running PVRPM tests with pysam <3.0 (thinking something went wrong with pysam 3.0) and still ran into the same errors. I also uninstalled everything and re-installed pysam from conda but still ran into the same issues. (I thought something might not have gone right with pip install - it's known to create problems sometimes).

I think maybe those who use PySAM don't use macOS and that it just doesn't work on that OS.

I have used PySAM on macOS in the past, and it worked without any issues.

brandons209 commented 2 years ago

Yeah the branch I am using is called macos-test you can pull it and modify it.

I really have no clue then on why that's occurring then. It seems the simulations work but not pulling a value from the modules, but then it works when putting a logging statement before and after the line.

If you have some old code using PySAM, could you try using the value method on one of the modules to pull out values? Cause that seems to be where the issue is occurring in PVRPM.

brandons209 commented 2 years ago

Maybe it is because a value doesn't exist in a PySAM module and it crashes there? In PVRPM I just made a generic function for getting values that may be what's causing the issue too.