search

alliander-opensource / weather-provider-api

This API is intended to help you fetch weather data from different data sources in an efficient and uniform way. By just supplying a list of locations and a time window you can get data for a specific source immediately. This project is licensed under the MPL-2.0 license.
https://alliander-opensource.github.io/weather-provider-api/
Mozilla Public License 2.0
27 stars 6 forks source link

Pygrib error in requirements.txt #9

Closed Ultiseeker closed 3 years ago

Ultiseeker commented 3 years ago

cloned the repo and run the requirments.txt on a windows machine with python 3.9.2 in a venv. got an error on the pygrib library when building the wheel. Is there an option to got this working?

machine i'm using:

Error message I'm getting:

Building wheels for collected packages: pygrib
  Building wheel for pygrib (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\users\user\appdata\local\programs\python\python39\python.exe' 'c:\users\user\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\user\AppData\Local\Temp\tmp6xs9yxku'
       cwd: C:\Users\user\AppData\Local\Temp\pip-install-zj4ok7qn\pygrib_ba5bdd98d0974d72b6da402f935163e7
  Complete output (21 lines):
  eccodes not found, build may fail...
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.9
  creating build\lib.win-amd64-3.9\pygrib
  copying pygrib\__init__.py -> build\lib.win-amd64-3.9\pygrib
  running build_ext
  cythoning pygrib/_pygrib.pyx to pygrib\_pygrib.c
  C:\Users\user\AppData\Local\Temp\pip-build-env-9hp1mogx\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\user\AppData\Local\Temp\pip-install-zj4ok7qn\pygrib_ba5bdd98d0974d72b6da402f935163e7\pygrib\_pygrib.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)
  building 'pygrib._pygrib' extension
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  creating build\temp.win-amd64-3.9\Release\pygrib
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\user\appdata\local\programs\python\python39\include -Ic:\users\user\appdata\local\programs\python\python39\include -IC:\Users\user\AppData\Local\Temp\pip-build-env-9hp1mogx\overlay\Lib\site-packages\numpy\core\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29910\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /Tcpygrib\_pygrib.c /Fobuild\temp.win-amd64-3.9\Release\pygrib\_pygrib.obj
  _pygrib.c
  C:\Users\user\AppData\Local\Temp\pip-build-env-9hp1mogx\overlay\Lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
  pygrib\_pygrib.c(620): fatal error C1083: Cannot open include file: 'grib_api.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for pygrib
Failed to build pygrib
ERROR: Could not build wheels for pygrib which use PEP 517 and cannot be installed directly
rflinnenbank commented 3 years ago

Pygrib and PEP 517 - (As far as we know) a Windows specific problem

First of all, thank you for notifying us on this. It is actually a known problem related to pygrib itself, and only occurs under certain environments. The pygrib build uses PEP 517 and as such apparently cannot be constructed as a wheel like the other affiliated packages. Under Linux this doesn't form much of a problem as the readily available make tools automatically will take over and construct the package as it was intended anyway.

Under Windows however, this functionality isn't readily available, meaning that one would have to either use an environment management system that can handle the situation (like Conda) or install the required tooling to build pygrib first.

https://visualstudio.microsoft.com/downloads/

Going to the above link and installing 'Build Tools for Visual Studio 2019' should get you the tooling required for the package te build from pip. (using the --no-use-pep517 parameter is advised to prevent further nagging about PEP 517)

Future solutions

We, of course, want the Weather Provider API to be as simple to use as possible and the use of pygrib currently also means some extra configuration steps on certain server configurations. Which means we really want to phase out pygrib in favor of cfgrib, which doesn't bring all of these things to the table.

As of right now the sole reason for using the pygrib package is the Harmonie Arome model, which is based on the parsing of a proprietary format GRIB file. We hope to be able to transform this handling into a cfgrib-compatible format pretty soon, which should immediately allow us to purge pygrib and issue arising from its use.

Ultiseeker commented 3 years ago

Do I need to use visual studio code 2019 with the build tools? because i'm still getting an error with powershell This is after I've installed visual studio 2019 and the Build tools

rflinnenbank commented 3 years ago

You shouldn't need VSC 2019, but the basic build commands (like nmake or cl) should be accessible from the command-line environment you're using pip from. (this may mean having to run 'vcvarsall.bat' or a variant thereof)

I myself use PyCharm for project maintenance on Windows (and Linux for the actual server to run it on) and I find that restarting PyCharm is usually more then enough to propagate any environment settings that have been added since the previous startup if they don't work already.

Because we want users to be able to use the project without spending hours of configuring them I'll be sure to spend some time upcoming Friday (CET over here) to document the full process for setting up the Build Tools while doing a clean setup. Ideally I'll also have pygrib out of the project before the end of next week, but that isn't something I can guarantee yet, unfortunately..

rflinnenbank commented 3 years ago

From what I get it's probably easier to use miniconda or anaconda unless there is a reason not to. Both can install pygrib for you after which pip can easily install the remainder. This is of course still a pretty messy situation, and only confirms that it would be best to get rid of pygrib altogether.

Fortunately now that pygrib has allowed us to analyze and deconstruct the Harmonie Arome dataset, we should be able to use the functionality of cfgrib to get the same results..

rflinnenbank commented 3 years ago

I think I may have still found a possible solution for your situation, though there are some for whom it doesn't work..

First of all, check your VS 2019 Build Tools installation by re-running the setup or changing the installation from Software. From there you should be able to modify the installation. Find the C++ Build Tools and add those to the installation if they weren't already. You may need to restart.

Then from the project environment use pip to install eccodes. To allow the pygrib installation to detect the eccodes installation, set the following environment variables: LDFLAGS='-L/my/libeccodes/path/lib' CFLAGS='-I/my/libeccodes/path/lib'

(With '/my/libeccodes/path/lib' being the lib folder within the eccodes installation) This should allow the build tools to find the grib_api.h file and compile properly.

Once pygrib is properly compiled, the rest shouldn't be any problem anymore...

rflinnenbank commented 3 years ago

In between status update: Thanks to the sledgehammer functionality that pygrib offers we were able to break open and view the inside of the Harmonie Arome GRIB format . And now that we know what can be found, how and where, we can use cfgrib to more delicately pluck the same data from the file.

Preliminary tests with cfgrib show that we can now easily gather the same data as before without having to read line by line and manually constructing an Xarray Dataset from the results. After testing and confirming the contents this will be worked into an update for the current method, immediately removing the pygrib package from the current setup.

rflinnenbank commented 3 years ago

The cfgrib method works perfectly, but a bit too much so for now. To match the Weather API's own (up to) 2.0 formatting we'll need to flatten the index of certain fields. We simultaneously also want to check out the used indexation for use in a currently being designed 3.0 format.

rflinnenbank commented 3 years ago

Fully replaced the original pygrib method for Arome with a cfgrib based implementation. With pygrib now out of the picture, all the issues that came with it should be solved as well.