pypi idaes-pse[ui] does not work with idaes-pse from GitHub main

dangunter commented 10 months ago

@lbianchi-lbl note that pypi idaes-ui + pypi idaes-pse seems ok (no need to take down package), and both from GH also seems OK.

Proposed solution: release the current code in idaes-ui/main as a new pypi package (and then deprecate the old one, if possible).

To reproduce:

clone and install idaes-pse from GitHub main (pip install -r requirements-dev.txt)
Install idaes-ui from pip with: pip install idaes[ui]
Install idaes-examples (pip install idaes-examples)
Open an example notebook (e.g. through "jupyter lab" started in a home dir) and in the appropriate place run visualize()

Behavior:

Exception occurred during processing of request from ('127.0.0.1', 51102)
Traceback (most recent call last):
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 270, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 671, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 306, in do_GET
    self._get_app(id_)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 326, in _get_app
    with open(p, "r", encoding="utf-8") as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dkgun\\miniconda3\\envs\\idaes-pse-py311\\Lib\\site-packages\\idaes_ui\\fv\\..\\..\\IDAES-UI\\dist\\index.html'
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 51103)
Traceback (most recent call last):
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 270, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 671, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 306, in do_GET
    self._get_app(id_)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 326, in _get_app
    with open(p, "r", encoding="utf-8") as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dkgun\\miniconda3\\envs\\idaes-pse-py311\\Lib\\site-packages\\idaes_ui\\fv\\..\\..\\IDAES-UI\\dist\\index.html'
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 51104)
Traceback (most recent call last):
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 270, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 671, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 306, in do_GET
    self._get_app(id_)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 326, in _get_app
    with open(p, "r", encoding="utf-8") as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dkgun\\miniconda3\\envs\\idaes-pse-py311\\Lib\\site-packages\\idaes_ui\\fv\\..\\..\\IDAES-UI\\dist\\index.html'
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 51108)
Traceback (most recent call last):
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 270, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 671, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 306, in do_GET
    self._get_app(id_)
  File "C:\Users\dkgun\miniconda3\envs\idaes-pse-py311\Lib\site-packages\idaes_ui\fv\model_server.py", line 326, in _get_app
    with open(p, "r", encoding="utf-8") as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dkgun\\miniconda3\\envs\\idaes-pse-py311\\Lib\\site-packages\\idaes_ui\\fv\\..\\..\\IDAES-UI\\dist\\index.html'
----------------------------------------

lbianchi-lbl commented 10 months ago

@dangunter can you check the second bullet in the "Steps to reproduce" list? pip install idaes[ui] wouldn't work so I guess this could be either:

(A) pip install idaes-ui
(B) pip install idaes-pse[ui] (which seems less likely, since idaes-pse would have already been installed in Step 1)

lbianchi-lbl commented 10 months ago

I confirm I can reproduce the error with the following steps:

Run commands:

mkdir idaes-ui-36 && cd idaes-ui-36
conda create --yes --name test-idaes-ui-36 python=3.11 && conda activate test-idaes-ui-36
git clone https://github.com/IDAES/idaes-pse && cd idaes-pse && pip install -r requirements-dev.txt && cd -
pip install idaes-ui  # this didn't install any new package
pip install "idaes-examples[gui,docs]"  # adding [docs] is needed or a ModuleNotFound error will be raised
idaesx gui

Select docs/tut/ui in the GUI and click "Open"
Run cells in sequence (the error occurs in Cell 3)

This is the output of pip list in that environment:

Package                           Version      Editable project location
--------------------------------- ------------ -----------------------------
absl-py                           2.0.0
addheader                         0.3.2
alabaster                         0.7.13
anyio                             4.0.0
argon2-cffi                       23.1.0
argon2-cffi-bindings              21.2.0
arrow                             1.3.0
astroid                           2.9.3
asttokens                         2.4.1
astunparse                        1.6.3
async-lru                         2.0.4
attrs                             21.4.0
Babel                             2.13.1
beautifulsoup4                    4.12.2
black                             22.3.0
bleach                            6.1.0
cachetools                        5.3.2
certifi                           2023.7.22
cffi                              1.16.0
cfgv                              3.4.0
charset-normalizer                3.3.2
click                             8.1.7
colorama                          0.4.6
comm                              0.2.0
contourpy                         1.2.0
CoolProp                          6.5.0.post1
coramin                           0.1.1
coverage                          7.3.2
cycler                            0.12.1
debugpy                           1.8.0
decorator                         5.1.1
defusedxml                        0.7.1
distlib                           0.3.7
docutils                          0.17.1
entrypoints                       0.4
et-xmlfile                        1.1.0
executing                         2.0.1
fastjsonschema                    2.18.1
filelock                          3.13.1
flake8                            4.0.1
flatbuffers                       23.5.26
fonttools                         4.44.1
fqdn                              1.5.1
gast                              0.5.4
gitdb                             4.0.11
GitPython                         3.1.40
google-auth                       2.23.4
google-auth-oauthlib              1.1.0
google-pasta                      0.2.0
greenlet                          3.0.1
gridx-egret                       0.5.5
gridx-prescient                   2.2.2
grpcio                            1.59.2
h5py                              3.10.0
idaes-examples                    2.2.0
idaes-pse                         2.3.0.dev0   /tmp/tmp.2rlNYlyzGy/idaes-pse
idaes-ui                          0.23.8
identify                          2.5.31
idna                              3.4
imagesize                         1.4.1
importlib-metadata                6.8.0
iniconfig                         2.0.0
ipykernel                         6.26.0
ipython                           8.17.2
ipython-genutils                  0.2.0
ipywidgets                        7.8.1
isoduration                       20.11.0
isort                             5.12.0
jedi                              0.19.1
Jinja2                            3.1.2
json5                             0.9.14
jsonpointer                       2.4
jsonschema                        4.17.3
jsonschema-specifications         2023.7.1
jupyter                           1.0.0
jupyter-book                      0.13.2
jupyter-cache                     0.4.3
jupyter_client                    8.6.0
jupyter-console                   6.6.3
jupyter-contrib-core              0.4.2
jupyter-contrib-nbextensions      0.7.0
jupyter_core                      5.5.0
jupyter-events                    0.6.3
jupyter-highlight-selected-word   0.2.0
jupyter-lsp                       2.2.0
jupyter-nbextensions-configurator 0.6.3
jupyter_server                    2.10.0
jupyter-server-mathjax            0.2.6
jupyter_server_terminals          0.4.4
jupyter-sphinx                    0.3.2
jupyterlab                        4.0.8
jupyterlab-pygments               0.2.2
jupyterlab_server                 2.24.0
jupyterlab-widgets                1.1.7
keras                             2.15.0
kiwisolver                        1.4.5
latexcodec                        2.0.1
lazy-object-proxy                 1.9.0
libclang                          16.0.6
linkify-it-py                     2.0.2
lxml                              4.9.3
Markdown                          3.4.4
markdown-it-py                    1.1.0
MarkupSafe                        2.1.3
matplotlib                        3.8.1
matplotlib-inline                 0.1.6
mccabe                            0.6.1
mdit-py-plugins                   0.2.8
mistune                           0.8.4
ml-dtypes                         0.2.0
mpmath                            1.3.0
mypy-extensions                   1.0.0
myst-nb                           0.13.2
myst-parser                       0.15.2
nbclient                          0.5.13
nbconvert                         6.5.4
nbdime                            3.2.1
nbformat                          5.9.2
nest-asyncio                      1.5.8
networkx                          3.2.1
nodeenv                           1.8.0
notebook                          7.0.6
notebook_shim                     0.2.3
numpy                             1.26.2
oauthlib                          3.2.2
omlt                              1.1
openpyxl                          3.1.2
opt-einsum                        3.3.0
overrides                         7.4.0
packaging                         23.2
pandas                            2.1.3
pandocfilters                     1.5.0
parso                             0.8.3
pathspec                          0.11.2
pexpect                           4.8.0
Pillow                            9.5.0
Pint                              0.22
pip                               23.3
platformdirs                      3.11.0
pluggy                            1.3.0
ply                               3.11
pockets                           0.9.1
pre-commit                        3.5.0
prometheus-client                 0.18.0
prompt-toolkit                    3.0.41
protobuf                          4.23.4
psutil                            5.9.6
ptyprocess                        0.7.0
pure-eval                         0.2.2
pyasn1                            0.5.0
pyasn1-modules                    0.3.0
pybtex                            0.24.0
pybtex-docutils                   1.0.3
pycodestyle                       2.8.0
pycparser                         2.21
pydata-sphinx-theme               0.8.1
pyflakes                          2.4.0
Pygments                          2.16.1
pylint                            2.12.2
Pyomo                             6.6.2
pyparsing                         3.1.1
PyPDF2                            3.0.1
pyrsistent                        0.20.0
PySimpleGUI                       4.60.5
pytest                            7.4.3
pytest-cov                        4.1.0
python-dateutil                   2.8.2
python-json-logger                2.0.7
pytz                              2023.3.post1
PyYAML                            6.0.1
pyzmq                             25.1.1
qtconsole                         5.5.0
QtPy                              2.4.1
referencing                       0.31.0
requests                          2.31.0
requests-oauthlib                 1.3.1
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rpds-py                           0.12.0
rsa                               4.9
scipy                             1.11.3
seaborn                           0.13.0
Send2Trash                        1.8.2
setuptools                        68.0.0
six                               1.16.0
smmap                             5.0.1
sniffio                           1.3.0
snowballstemmer                   1.2.1
soupsieve                         2.5
Sphinx                            4.5.0
sphinx-argparse                   0.4.0
sphinx-book-theme                 0.3.3
sphinx-comments                   0.0.3
sphinx-copybutton                 0.5.2
sphinx_design                     0.1.0
sphinx_external_toc               0.3.1
sphinx-jupyterbook-latex          0.5.2
sphinx-multitoc-numbering         0.1.3
sphinx-rtd-theme                  1.0.0
sphinx-thebe                      0.2.1
sphinx-togglebutton               0.3.2
sphinxcontrib-applehelp           1.0.4
sphinxcontrib-bibtex              2.5.0
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.1
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-napoleon            0.7
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
SQLAlchemy                        1.4.50
stack-data                        0.6.3
sympy                             1.12
tensorboard                       2.15.1
tensorboard-data-server           0.7.2
tensorflow                        2.15.0
tensorflow-estimator              2.15.0
tensorflow-io-gcs-filesystem      0.34.0
termcolor                         2.3.0
terminado                         0.18.0
tinycss2                          1.2.1
tinydb                            4.8.0
tkhtmlview                        0.1.1.post5
toml                              0.10.2
tornado                           6.3.3
traitlets                         5.13.0
types-python-dateutil             2.8.19.14
typing_extensions                 4.8.0
tzdata                            2023.3
uc-micro-py                       1.0.2
uri-template                      1.3.0
urllib3                           2.1.0
virtualenv                        20.24.6
wcwidth                           0.2.10
webcolors                         1.13
webencodings                      0.5.1
websocket-client                  1.6.4
Werkzeug                          3.0.1
wheel                             0.41.2
widgetsnbextension                3.6.6
wrapt                             1.13.3
xlrd                              2.0.1
zipp                              3.17.0

lbianchi-lbl commented 10 months ago

Looking at the directory tree of the idaes_ui package after it gets installed in non-editable mode (i.e. in a Conda env with Python 3.11, under $CONDA_PREFIX/lib/python3.11/site-packages/idaes_ui), it looks like the error is caused by the fact that the IDAES-UI directory tree does not exist.

I'm not familiar with the details, but it looks this contains only JS/HTML/CSS assets, i.e. no .py files. Since this is a top-level directory in the repository, it's not surprising that it wouldn't get included when building the package distribution (and/or when installing in non-editable mode).

(I'll update with possible remedies later)

lbianchi-lbl commented 10 months ago

Summarizing the next steps:

[ ] Remove the code currently in idaes_ui/fv/static (this is the "old", pre-React version, which we don't need to keep around)
[ ] Move code that's currently under IDAES-UI/dist to idaes_ui/fv/static
[ ] Change https://github.com/IDAES/idaes-ui/blob/cc1982368ee5a015754ac9b019aa5a63562d2eb2/idaes_ui/fv/model_server.py#L49 so that it points to idaes_ui.fv.static
[ ] Configure pyproject.toml so that package_data from idaes_ui/fv/static is included when building sdist/bdist_wheel
- As a shortcut, we can enable setuptools_scm so that all package files that are checked into the Git repo are automatically copied over (without having to specify all needed patterns in package_data)

lbianchi-lbl commented 9 months ago

This should be fixed by #39 and part of the 0.23.12 release.

IDAES / idaes-ui

pypi idaes-pse[ui] does not work with idaes-pse from GitHub main #36