Cannot use R models in version 0.11.x--follow up

bertbesser commented 1 year ago

Description

This is a follow-up bug report to #2414.

Problem: Some *.R files are missing when installing gluonts 0.11.x via poetry.

To Reproduce

I reproduced like this:

fresh poetry projects for both gluonts 0.9.x (the version we're using right now) and gluonts 0.11.x (the one we want to use)
both use python 3.8.x and poetry 1.2.x (i'm using a poetry beta, a team mate has the same issue with the most recent poetry version 1.2.2)

Result:

*.R files are there for 0.9.x and none is there for 0.11.x
both versions have a folder of *.R files, see here for 11 and here for 9

I provide two code blocks (of one shell sessions) how to reproduce.

/tmp> python --version
Python 3.8.10
/tmp> poetry --version
Poetry (version 1.2.0b1)
/tmp> mkdir gluonts11
/tmp> mkdir gluonts9
/tmp> cd gluonts11 
/tmp/gluonts11> echo "[virtualenvs]\nin-project = true" >> poetry.toml
/tmp/gluonts11> cat poetry.toml 
[virtualenvs]
in-project = true
/tmp/gluonts11> poetry init

This command will guide you through creating your pyproject.toml config.

Package name [gluonts11]:  
Version [0.1.0]:  
Description []:  
Author [Bert Besser <...>, n to skip]:  
License []:  
Compatible Python versions [^3.8]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] 
You can specify a package in the following forms:
  - A single name (requests)
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Search for package to add (or leave blank to continue): gluonts@^0.11.2
Adding gluonts@^0.11.2

Add a package: 

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "gluonts11"
version = "0.1.0"
description = ""
authors = ["Bert Besser <...>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"
gluonts = "^0.11.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes] 
/tmp/gluonts11> poetry install
Creating virtualenv gluonts11 in /tmp/gluonts11/.venv
Updating dependencies
Resolving dependencies... (0.6s)

Writing lock file

Package operations: 10 installs, 0 updates, 0 removals

  • Installing six (1.16.0)
  • Installing numpy (1.23.5)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.6)
  • Installing typing-extensions (4.4.0)
  • Installing pandas (1.5.2)
  • Installing pydantic (1.10.2)
  • Installing toolz (0.12.0)
  • Installing tqdm (4.64.1)
  • Installing gluonts (0.11.2)
/tmp/gluonts11> find .venv | grep "\.R"
exit 1
/tmp/gluonts11> # no R files here for 0.11.x

So there are no *.R files for 0.11.x.

... continuing in the same shell session ...

/tmp/gluonts11> cd ../gluonts9
/tmp/gluonts9> echo "[virtualenvs]\nin-project = true" >> poetry.toml    
/tmp/gluonts9> poetry init

This command will guide you through creating your pyproject.toml config.

Package name [gluonts9]:  
Version [0.1.0]:  
Description []:  
Author [Bert Besser <...>, n to skip]:  
License []:  
Compatible Python versions [^3.8]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] 
You can specify a package in the following forms:
  - A single name (requests)
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Search for package to add (or leave blank to continue): gluonts@^0.9.4
Adding gluonts@^0.9.4

Add a package: 

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "gluonts9"
version = "0.1.0"
description = ""
authors = ["Bert Besser <...>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"
gluonts = "^0.9.4"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes] 
/tmp/gluonts9> poetry install
Creating virtualenv gluonts9 in /tmp/gluonts9/.venv
Updating dependencies
Resolving dependencies... (1.3s)

Writing lock file

Package operations: 25 installs, 1 update, 0 removals

  • Installing pyparsing (3.0.9)
  • Installing numpy (1.23.5)
  • Installing packaging (21.3)
  • Installing pymeeus (0.5.11)
  • Updating setuptools (65.3.0 -> 65.6.0)
  • Installing six (1.16.0)
  • Installing tomli (2.0.1)
  • Installing typing-extensions (4.4.0)
  • Installing contourpy (1.0.6): Installing...
  • Installing contourpy (1.0.6)
  • Installing convertdate (2.4.0)
  • Installing cycler (0.11.0)
  • Installing fonttools (4.38.0)
  • Installing hijri-converter (2.2.4)
  • Installing kiwisolver (1.4.4)
  • Installing korean-lunar-calendar (0.3.1)
  • Installing pillow (9.3.0)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.6)
  • Installing setuptools-scm (7.0.5)
  • Installing holidays (0.17)
  • Installing matplotlib (3.6.2)
  • Installing pandas (1.5.2)
  • Installing pydantic (1.10.2)
  • Installing toolz (0.12.0)
  • Installing tqdm (4.64.1)
  • Installing gluonts (0.9.9)
/tmp/gluonts9> find .venv | grep "\.R"                               
.venv/lib/python3.8/site-packages/gluonts/model/r_forecast/R/forecast_methods.R
tmp/gluonts9> # for 0.9.x there is an R file

So there is an *.R file for 0.9.x.

Error message or code output

I will ask my colleague to paste some error message in the following comments.

Environment

Operating system: linux
Python version: 3.8.x
GluonTS version: 0.11.x resp. 0.9.x
MXNet version: ?

lostella commented 1 year ago

Thanks @bertbesser, I could reproduce this with pip as well, so this is something that needs to be fixed

lostella commented 1 year ago

This is the diff in setup.py between v0.10.x and v0.11.x

diff --git a/setup.py b/setup.py
index 3fd519e2..5795034f 100644
--- a/setup.py
+++ b/setup.py
@@ -63,7 +63,9 @@ def get_version_and_cmdclass(version_file):
     return globals_["__version__"], globals_["cmdclass"]()

-version, version_cmdclass = get_version_and_cmdclass("src/gluonts/_version.py")
+version, version_cmdclass = get_version_and_cmdclass(
+    "src/gluonts/meta/_version.py"
+)

 class TypeCheckCommand(distutils.cmd.Command):
@@ -202,14 +204,11 @@ shell_require = find_requirements("requirements-extras-shell.txt")
 mxnet_require = find_requirements("requirements-mxnet.txt")
 torch_require = find_requirements("requirements-pytorch.txt")

-setup_requires = find_requirements("requirements-setup.txt")
-
 dev_require = (
     arrow_require
     + docs_require
     + tests_require
     + shell_require
-    + setup_requires
     + sagemaker_api_require
 )

@@ -235,7 +234,6 @@ setup_kwargs: dict = dict(
     package_dir={"": "src"},
     packages=find_namespace_packages(include=["gluonts*"], where=str(SRC)),
     include_package_data=True,
-    setup_requires=setup_requires,
     install_requires=find_requirements("requirements.txt"),
     tests_require=tests_require,
     extras_require={
@@ -249,48 +247,16 @@ setup_kwargs: dict = dict(
         "shell": shell_require,
         "torch": torch_require,
     },
-    entry_points={
-        "gluonts_forecasters": [
-            "deepar=gluonts.model.deepar:DeepAREstimator",
-            "DeepAR=gluonts.model.deepar:DeepAREstimator",
-            "DeepFactor=gluonts.model.deep_factor:DeepFactorEstimator",
-            "DeepState=gluonts.model.deepstate:DeepStateEstimator",
-            "DeepVAR=gluonts.model.deepvar:DeepVAREstimator",
-            "GaussianProcess=gluonts.model.gp_forecaster:GaussianProcessEstimator",
-            "GPVAR=gluonts.model.gpvar:GPVAREstimator",
-            "LSTNet=gluonts.model.lstnet:LSTNetEstimator",
-            "NBEATS=gluonts.model.n_beats:NBEATSEstimator",
-            "NBEATSEnsemble=gluonts.model.n_beats:NBEATSEnsembleEstimator",
-            "NPTS=gluonts.model.npts:NPTSPredictor",
-            "Rotbaum=gluonts.model.rotbaum:TreeEstimator",
-            "SelfAttention=gluonts.model.san:SelfAttentionEstimator",
-            "SeasonalNaive=gluonts.model.seasonal_naive:SeasonalNaivePredictor",
-            "MQCNN=gluonts.model.seq2seq:MQCNNEstimator",
-            "MQRNN=gluonts.model.seq2seq:MQRNNEstimator",
-            "Seq2Seq=gluonts.model.seq2seq:Seq2SeqEstimator",
-            "SimpleFeedForward=gluonts.model.simple_feedforward:SimpleFeedForwardEstimator",
-            "TFT=gluonts.model.tft:TemporalFusionTransformerEstimator",
-            "DeepTPP=gluonts.model.tpp:DeepTPPEstimator",
-            "Transformer=gluonts.model.transformer:TransformerEstimator",
-            "Constant=gluonts.model.trivial.constant:ConstantPredictor",
-            "ConstantValue=gluonts.model.trivial.constant:ConstantValuePredictor",
-            "Identity=gluonts.model.trivial.identity:IdentityPredictor",
-            "Mean=gluonts.model.trivial.mean:MeanEstimator",
-            "MeanPredictor=gluonts.model.trivial.mean:MeanPredictor",
-            "MovingAverage=gluonts.model.trivial.mean:MovingAveragePredictor",
-            "WaveNet=gluonts.model.wavenet:WaveNetEstimator",
-            # "r=gluonts.model.r_forecast:RForecastPredictor [R]",
-            # "prophet=gluonts.model.prophet:ProphetPredictor [Prophet]",
-        ],
-        "pygments.styles": [
-            "gluonts-dark=gluonts.meta.style:Dark",
-        ]
-    },
     cmdclass={
         "type_check": TypeCheckCommand,
         "style_check": StyleCheckCommand,
         **version_cmdclass,
     },
+    entry_points={
+        "pygments.styles": [
+            "gluonts-dark=gluonts.meta.style:Dark",
+        ]
+    },
 )

 if HAS_SPHINX:

jmberutich commented 1 year ago

I have also exactly the same issue. R folder is missing in 0.11 install via pip

lostella commented 1 year ago

This is extremely weird: if I do

python setup.py sdist bdist_wheel

at a5ba84a41a27def5a5108c8a31ee949c15430d63 (the v0.11.0 tag), the resulting archive doesn't have the R files; but if I do it at 8a8bed57eae93509afb646ccab7434ad2bb2f4fd (which is the commit immediately before) the R files are there. And the diff between the two commits is empty!

lostella commented 1 year ago

Package publishing workflow runs:

The latter does not create the R sub-folder, and does not copy its content: does it have it the first place?

lostella commented 1 year ago

@jaheba I believe this may require the src/ prefix? See https://packaging.python.org/en/latest/guides/using-manifest-in/#manifest-in-commands

bertbesser commented 1 year ago

I was also just browsing this section of the code. It does not have the src/ prefix on the 0.9.x and 0.10.x branches/tags...

bertbesser commented 1 year ago

Unless there's other changes that change the source root. However, this section of the setup.py is unchanged:

     package_dir={"": "src"},
     packages=find_namespace_packages(include=["gluonts*"], where=str(SRC)),

bertbesser commented 1 year ago

Hi I tested via poetry add "git+https://github.com/lostella/gluon-ts.git#5d08ce" from your fork @lostella and the .R files were there. I see you have already merged it! 🙏

This was quick! Do you already have an idea when this will be available in the 0.11.x branch, or even published to pypi? (Asking to arrange our time/prios in the project.)

lostella commented 1 year ago

This was quick! Do you already have an idea when this will be available in the 0.11.x branch, or even published to pypi? (Asking to arrange our time/prios in the project.)

Nice! Thanks for trying it out; we're doing a 0.11.3 bugfix release today.

lostella commented 1 year ago

@bertbesser v0.11.3 was released, and R based models should now work again. Could I ask you to double-check?

bertbesser commented 1 year ago

Hi, sure. I'm not personally the one who can do the double check. But I will make sure that we'll report back asap.

jmberutich commented 1 year ago

This was quick! Do you already have an idea when this will be available in the 0.11.x branch, or even published to pypi? (Asking to arrange our time/prios in the project.)

Nice! Thanks for trying it out; we're doing a 0.11.3 bugfix release today.

Awesome, thanks

bertbesser commented 1 year ago

Hi @lostella. Reporting back: It works. => Solved for us.

Thank you for fixing this so promptly! Saves us a good amout of hassle 😄

melopeo commented 1 year ago

Hi,

I have done a couple of checks for univariate models and hierarchical models, and now it seems to be in order.

Lets close this issue.

Thank you all!

awslabs / gluonts