glitch401
commented
2 months ago can I take this issue up?
Open peterdudfield opened 2 months ago
glitch401
commented
2 months ago can I take this issue up?
peterdudfield
commented
2 months ago yes please thanks @glitch401
glitch401
commented
2 months ago apologies for circling back on this issue so late, but in the given resources to get this project running. I'm facing issues in successfully testing this library:
After installation of requirements as mentioned in the Readme:
conda install python=3.10
conda install -c conda-forge xesmf esmpy h5py -y
pip install torch==2.2.0 --index-url https://download.pytorch.org/whl/cpu
pip install git+https://github.com/SheffieldSolar/PV_Live-API#pvlive_api
pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cpu
pip install -e .I'm getting the following error on running pytest:
ERROR tests/test_app.py::test_app - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
ERROR tests/test_app.py::test_app_day_ahead_model - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))@peterdudfield what am i missing, can you help me navigate this?
peterdudfield
commented
2 months ago I think you need to have docker server running locally aswell
glitch401
commented
2 months ago makes sense, will try running it
glitch401
commented
2 months ago just curious if I'm running the docker right:
(.conda) neil@Indranils-MBP pvnet_app % docker run -p 8000:8000 pvnet_app
/opt/conda/lib/python3.10/site-packages/pydantic/_internal/_config.py:321: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'orm_mode' has been renamed to 'from_attributes'
* 'underscore_attrs_are_private' has been removed
warnings.warn(message, UserWarning)
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:214} INFO - Using `pvnet` library version: 3.0.47
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:215} INFO - Using `pvnet_app` library version: 2.3.12
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:216} INFO - Using 10 workers
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:223} INFO - Using adjduster: True
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:224} INFO - Saving GSP sum: False
[2024-07-20 01:47:05,376] {/app/pvnet_app/app.py:239} INFO - Making forecast for init time: 2024-07-20 01:30:00
[2024-07-20 01:47:05,377] {/app/pvnet_app/app.py:240} INFO - Making forecast for GSP IDs: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317]
[2024-07-20 01:47:05,377] {/app/pvnet_app/app.py:246} INFO - Loading GSP metadata
[2024-07-20 01:47:05,377] {/opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:133} INFO - Loading GSP data from database
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /app/pvnet_app/app.py:248 in app │
│ │
│ 245 │ # Make pands Series of most recent GSP effective capacities │
│ 246 │ logger.info("Loading GSP metadata") │
│ 247 │ │
│ ❱ 248 │ ds_gsp = next(iter(OpenGSPFromDatabase())) │
│ 249 │ │
│ 250 │ # Get capacities from the database │
│ 251 │ db_connection = DatabaseConnection(url=os.getenv("DB_URL"), base=B │
│ │
│ ╭───────────────────────────── locals ─────────────────────────────╮ │
│ │ day_ahead_model_used = False │ │
│ │ gsp_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ... +307] │ │
│ │ num_workers = 10 │ │
│ │ t0 = Timestamp('2024-07-20 01:30:00') │ │
│ │ temp_dir = <TemporaryDirectory '/tmp/tmpt72lhax6'> │ │
│ │ write_predictions = True │ │
│ ╰──────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/torch/utils/data/datapipes/_hook_ite │
│ rator.py:181 in wrap_generator │
│ │
│ 178 │ │ │ │ │ with profiler_record_fn_context(datapipe): │
│ 179 │ │ │ │ │ │ response = gen.send(None) │
│ 180 │ │ │ │ else: │
│ ❱ 181 │ │ │ │ │ response = gen.send(None) │
│ 182 │ │ │ │ │
│ 183 │ │ │ │ while True: │
│ 184 │ │ │ │ │ datapipe._number_of_samples_yielded += 1 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ _profiler_enabled = False │ │
│ │ args = (OpenGSPFromDatabaseIterDataPipe,) │ │
│ │ datapipe = OpenGSPFromDatabaseIterDataPipe │ │
│ │ full_msg = 'thrown by __iter__ of │ │
│ │ OpenGSPFromDatabaseIterDataPipe(interpolat… │ │
│ │ lo'+41 │ │
│ │ func = <function │ │
│ │ OpenGSPFromDatabaseIterDataPipe.__iter__ at │ │
│ │ 0xffff2221c790> │ │
│ │ gen = <generator object │ │
│ │ OpenGSPFromDatabaseIterDataPipe.__iter__ at │ │
│ │ 0xffff01a49850> │ │
│ │ iterator_id = 0 │ │
│ │ kwargs = {} │ │
│ │ msg = 'thrown by __iter__ of' │ │
│ │ profiler_record_fn_context = <function │ │
│ │ hook_iterator.<locals>.profiler_record_fn_… │ │
│ │ at 0xffff2221c820> │ │
│ │ single_iterator_msg = 'single iterator per IterDataPipe │ │
│ │ constraint' │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:7 │
│ 3 in __iter__ │
│ │
│ 70 │ │ │ gsp_pv_power_mw_df, │
│ 71 │ │ │ gsp_nominal_capacity, │
│ 72 │ │ │ gsp_effective_capacity, │
│ ❱ 73 │ │ ) = get_gsp_power_from_database( │
│ 74 │ │ │ history_duration=self.history_duration, │
│ 75 │ │ │ interpolate_minutes=self.interpolate_minutes, │
│ 76 │ │ │ load_extra_minutes=self.load_extra_minutes, │
│ │
│ ╭──────────────── locals ────────────────╮ │
│ │ self = OpenGSPFromDatabaseIterDataPipe │ │
│ ╰────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/ocf_datapipes/load/gsp/database.py:1 │
│ 55 in get_gsp_power_from_database │
│ │
│ 152 │ │
│ 153 │ # make database connection │
│ 154 │ url = os.getenv("DB_URL") │
│ ❱ 155 │ db_connection = DatabaseConnection(url=url, base=Base_Forecast) │
│ 156 │ │
│ 157 │ with db_connection.get_session() as session: │
│ 158 │ │ # We minus 1 second just to make sure we don't that value │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ empty_df = Empty DataFrame │ │
│ │ Columns: [] │ │
│ │ Index: [2024-07-19 23:00:00+00:00, 2024-07-19 │ │
│ │ 23:30:00+00:00, 2024-07-20 00:00:00+00:00, │ │
│ │ 2024-07-20 00:30:00+00:00, 2024-07-20 │ │
│ │ 01:00:00+00:00, 2024-07-20 01:30:00+00:00] │ │
│ │ extra_duration = datetime.timedelta(seconds=3600) │ │
│ │ gsp_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ... +307] │ │
│ │ history_duration = datetime.timedelta(seconds=5400) │ │
│ │ interpolate_minutes = 60 │ │
│ │ load_extra_minutes = 60 │ │
│ │ now = Timestamp('2024-07-20 01:30:00+0000', tz='UTC') │ │
│ │ start_utc = Timestamp('2024-07-20 00:00:00+0000', tz='UTC') │ │
│ │ start_utc_extra = Timestamp('2024-07-19 23:00:00+0000', tz='UTC') │ │
│ │ url = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/nowcasting_datamodel/connection.py:2 │
│ 7 in __init__ │
│ │
│ 24 │ │ self.url = url │
│ 25 │ │ self.base = base │
│ 26 │ │ │
│ ❱ 27 │ │ self.engine = create_engine(self.url, echo=echo) │
│ 28 │ │ │
│ 29 │ │ self.Session = sessionmaker(bind=self.engine) │
│ 30 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ base = <class 'sqlalchemy.orm.decl_api.Base'> │ │
│ │ echo = True │ │
│ │ self = <nowcasting_datamodel.connection.DatabaseConnection object at │ │
│ │ 0xffff01a6f940> │ │
│ │ url = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ in create_engine:2 │
│ ╭──────── locals ─────────╮ │
│ │ kwargs = {'echo': True} │ │
│ │ url = None │ │
│ ╰─────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py:281 │
│ in warned │
│ │
│ 278 │ │ │ │ │ │ version_warnings[m], │
│ 279 │ │ │ │ │ │ stacklevel=3, │
│ 280 │ │ │ │ │ ) │
│ ❱ 281 │ │ │ return fn(*args, **kwargs) # type: ignore[no-any-return] │
│ 282 │ │ │
│ 283 │ │ doc = fn.__doc__ is not None and fn.__doc__ or "" │
│ 284 │ │ if doc: │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ args = (None,) │ │
│ │ check_any_kw = 'kwargs' │ │
│ │ check_defaults = () │ │
│ │ check_kw = { │ │
│ │ │ 'implicit_returning', │ │
│ │ │ 'empty_in_strategy', │ │
│ │ │ 'strategy' │ │
│ │ } │ │
│ │ fn = <function create_engine at 0xffff476aa9e0> │ │
│ │ kwargs = {'echo': True} │ │
│ │ m = 'strategy' │ │
│ │ messages = { │ │
│ │ │ 'strategy': 'The create_engine.strategy keyword │ │
│ │ is deprecated, and the only argument accepted'+177, │ │
│ │ │ 'empty_in_strategy': 'The │ │
│ │ create_engine.empty_in_strategy keyword is │ │
│ │ deprecated, and no longer has any'+181, │ │
│ │ │ 'implicit_returning': 'The │ │
│ │ create_engine.implicit_returning parameter is │ │
│ │ deprecated and will be removed'+22 │ │
│ │ } │ │
│ │ version_warnings = { │ │
│ │ │ 'strategy': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'>, │ │
│ │ │ 'empty_in_strategy': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'>, │ │
│ │ │ 'implicit_returning': <class │ │
│ │ 'sqlalchemy.exc.SADeprecationWarning'> │ │
│ │ } │ │
│ │ versions = { │ │
│ │ │ 'strategy': '1.4', │ │
│ │ │ 'empty_in_strategy': '1.4', │ │
│ │ │ 'implicit_returning': '2.0' │ │
│ │ } │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/create.py:546 in │
│ create_engine │
│ │
│ 543 │ kwargs.pop("empty_in_strategy", None) │
│ 544 │ │
│ 545 │ # create url.URL object │
│ ❱ 546 │ u = _url.make_url(url) │
│ 547 │ │
│ 548 │ u, plugins, kwargs = u._instantiate_plugins(kwargs) │
│ 549 │
│ │
│ ╭──────── locals ─────────╮ │
│ │ kwargs = {'echo': True} │ │
│ │ url = None │ │
│ ╰─────────────────────────╯ │
│ │
│ /opt/conda/lib/python3.10/site-packages/sqlalchemy/engine/url.py:846 in │
│ make_url │
│ │
│ 843 │ elif not isinstance(name_or_url, URL) and not hasattr( │
│ 844 │ │ name_or_url, "_sqla_is_testing_if_this_is_a_mock_object" │
│ 845 │ ): │
│ ❱ 846 │ │ raise exc.ArgumentError( │
│ 847 │ │ │ f"Expected string or URL object, got {name_or_url!r}" │
│ 848 │ │ ) │
│ 849 │ else: │
│ │
│ ╭────── locals ──────╮ │
│ │ name_or_url = None │ │
│ ╰────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
ArgumentError: Expected string or URL object, got None
This exception is thrown by __iter__ of
OpenGSPFromDatabaseIterDataPipe(interpolate_minutes=60, load_extra_minutes=60,
national_only=False)Sorry you won't be able to run this app. You should be able to run the tests though?
glitch401
commented
1 month ago Gotcha. I was trying to run pytest
but there seems to be some disparity in the version of pandas
the environment that I've built is running 1.3.5, which has been automatically installed as a dependency to other packages.
but i'm facing the following error:
======================================================================= short test summary info ========================================================================
ERROR tests/test_app.py - AttributeError: module 'pandas.core.indexers' has no attribute 'objects'
ERROR tests/test_data.py - AttributeError: module 'pandas.core.indexers' has no attribute 'objects'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================== 23 warnings, 2 errors in 8.04s ====================================================================yea, i would try to use pandas 2.2.2, or the one just before 2.0.0
glitch401
commented
1 month ago is torch-geometric-temporal necessary for testing? coz, while I was trying to upgrade pandas to 1.5.3 which is the predecessor to 2.0.0 I'm faced with the error
ERROR: pip's dependency resolver dosent currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
torch-geometric-temporal 0.54.0 requires pandas<=1.3.5, but you have pandas 1.5.3 which is incompatible.
pvnet-summation 0.3.0 requires huggingface-hub==0.20.*, but you have huggingface-hub 0.23.5 which is incompatible.
pvnet-app 2.3.12 requires requests<2.32.0, but you have requests 2.32.3 which is incompatible. Can you give me a pip freeze of your env?
peterdudfield
commented
1 month ago please look in a recent CI, you'll be able to see the dependencies that were installed
glitch401
commented
1 month ago Was checking out the recent release and after creation of a fresh env, I'm faced with this:
__________________________________________________________ ERROR collecting tests/test_app.py ___________________________________________________________
ImportError while importing test module '/Users/neil/Documents/GitHub/pvnet_app/tests/test_app.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/util.py:8: in <module>
import esmpy as ESMF
E ModuleNotFoundError: No module named 'esmpy'
During handling of the above exception, another exception occurred:
../../../miniconda3/envs/pvnet_env/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_app.py:13: in <module>
from pvnet_app.data import sat_5_path, sat_15_path
pvnet_app/data.py:4: in <module>
import xesmf as xe
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/__init__.py:3: in <module>
from . import data, util
../../../miniconda3/envs/pvnet_env/lib/python3.10/site-packages/xesmf/util.py:10: in <module>
import ESMF
E ModuleNotFoundError: No module named 'ESMF'and my envs look like: envs.txt
peterdudfield
commented
1 month ago you might have to install with conda and do things like - https://github.com/openclimatefix/pvnet_app/blob/main/.github/workflows/conda-pytest.yaml#L25
glitch401
commented
1 month ago i was trying to avoid conda and get a hang on the dependencies and their versions. But this helped resolve the dependencies:
conda install -c conda-forge xesmf esmpy -y
pip install -e . there's a new error:
tests/test_app.py:13: in <module>
from pvnet_app.data import sat_5_path, sat_15_path
pvnet_app/data.py:10: in <module>
from ocf_datapipes.config.load import load_yaml_configuration
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/__init__.py:4: in <module>
import ocf_datapipes.convert
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/convert/__init__.py:3: in <module>
from .coordinates import ConvertGeostationaryToLonLatIterDataPipe as ConvertGeostationaryToLonLat
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/convert/coordinates.py:8: in <module>
from ocf_datapipes.utils.geospatial import (
../../../miniconda3/lib/python3.11/site-packages/ocf_datapipes/utils/geospatial.py:9: in <module>
import pvlib
../../../miniconda3/lib/python3.11/site-packages/pvlib/__init__.py:3: in <module>
from pvlib import ( # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/pvlib/spectrum/__init__.py:2: in <module>
from pvlib.spectrum.mismatch import ( # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/pvlib/spectrum/mismatch.py:12: in <module>
from scipy.interpolate import interp1d
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/__init__.py:167: in <module>
from ._interpolate import *
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:12: in <module>
from . import _fitpack_py
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_fitpack_py.py:8: in <module>
from ._fitpack_impl import bisplrep, bisplev, dblint # noqa: F401
../../../miniconda3/lib/python3.11/site-packages/scipy/interpolate/_fitpack_impl.py:103: in <module>
'iwrk': array([], dfitpack_int), 'u': array([], float),
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:46: in __repr__
arg_str = _construction_repr(dtype, include_align=False)
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:100: in _construction_repr
return _scalar_str(dtype, short=short)
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:143: in _scalar_str
elif np.issubdtype(dtype, np.number):
../../../miniconda3/lib/python3.11/site-packages/numpy/core/numerictypes.py:417: in issubdtype
arg1 = dtype(arg1).type
../../../miniconda3/lib/python3.11/site-packages/numpy/core/_dtype.py:46: in __repr__
arg_str = _construction_repr(dtype, include_align=False)
E RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)```any hints on this? @peterdudfield
peterdudfield
commented
1 month ago oh sorry for my slow reply, what version of numpy are you using? There has been a new release and it sometimes causes some issues
glitch401
commented
1 month ago Thats alright :D
I'm running on numpy 1.26.4
peterdudfield
commented
1 month ago Can you try numpy 2
glitch401
commented
1 month ago alright, new day new errors
So, from last time trying to solve the environment. I figured to pull the package versions from pvnet-app latest release which has upgraded the numpy and scipy and the environments looks like: envs.txt
the new error looks like something to do with the docker..: log.txt I recall from the previous comment, can I navigate through this?
peterdudfield
commented
1 month ago do you have docker running?
glitch401
commented
1 month ago docker as a service, yes
Client:
Version: 27.0.3
Context: desktop-linux
Debug Mode: falsebut the pvnet-app container, no
glitch401
commented
3 weeks ago docker as a service, yes
Client: Version: 27.0.3 Context: desktop-linux Debug Mode: falsebut the pvnet-app container, no
@peterdudfield ?
peterdudfield
commented
3 weeks ago sorry im not sure, are you still getting the same error?
glitch401
commented
3 weeks ago yeah getting the same thing. i'll try again removing everything and begin the process back from recloning
Detailed Description
Would be good to check the forecast is a physical solar forecast. This is tricky to define but there are some quick wins we can do
Context
Possible Implementation