search

stan-dev / pystan

PyStan, a Python interface to Stan, a platform for statistical modeling. Documentation: https://pystan.readthedocs.io
ISC License
342 stars 59 forks source link

building model results in ValueError: The JSON document has an improper structure... #350

Closed samhiggs closed 2 years ago

samhiggs commented 2 years ago

Describe the bug

Running any model definition results in a JSON document error as it's trying to build. I have tried using my model, the example model, a simple linear model all have the same result.

Trying to run pystan 3.4.0 in a linux/amd64 python container

Describe your system

Steps/Code to Reproduce

  1. Save the sample_stan.py file below in current path
  2. Start container docker run --platform linux/amd64 --rm -it -v $(pwd)/sample_stan.py:/sample_stan.py python:3.10-bullseye bash
  3. pip install pystan==3.4.0 (in container)
  4. python sample_stan.py (in container)

sample_stan.py

import stan

schools_code = """
data {
  int<lower=0> J;         // number of schools
  real y[J];              // estimated treatment effects
  real<lower=0> sigma[J]; // standard error of effect estimates
}
parameters {
  real mu;                // population treatment effect
  real<lower=0> tau;      // standard deviation in treatment effects
  vector[J] eta;          // unscaled deviation from mu by school
}
transformed parameters {
  vector[J] theta = mu + tau * eta;        // school treatment effects
}
model {
  target += normal_lpdf(eta | 0, 1);       // prior log-density
  target += normal_lpdf(y | theta, sigma); // log-likelihood
}
"""

schools_data = {"J": 8,
                "y": [28,  8, -3,  7, -1,  1, 18, 12],
                "sigma": [15, 10, 16, 11,  9, 11, 10, 18]}

posterior = stan.build(schools_code, data=schools_data)

Full Trace

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/httpstan/views.py", line 253, in handle_show_params
    services_module = httpstan.models.import_services_extension_module(model_name)
  File "/usr/local/lib/python3.10/site-packages/httpstan/models.py", line 90, in import_services_extension_module
    module: ModuleType = importlib.util.module_from_spec(spec)  # type: ignore
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /root/.cache/httpstan/4.7.2/models/f4plfhr5/stan_services_model_f4plfhr5.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/stan/model.py", line 517, in build
    return asyncio.run(go())
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/stan/model.py", line 509, in go
    raise RuntimeError(resp.json()["message"])
  File "/usr/local/lib/python3.10/site-packages/stan/common.py", line 24, in json
    return simdjson.loads(self.content)
  File "/usr/local/lib/python3.10/site-packages/simdjson/__init__.py", line 61, in loads
    return parser.parse(s, True)
ahartikainen commented 2 years ago

Can you try to build and install httpstan when creating the container.

I think something in httpstan is precompiled, then linked but then fails if used.

samhiggs commented 2 years ago

Yep so tried to both install from source and build.

Attempt 1

Install from source git clone ... && cd httpstan && git checkout 4.7.2 && pip install . resulted in no module named setuptools

Attempt 1 Full Trace

  × Building wheel for httpstan (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      A setup.py file already exists. Using it.
      Traceback (most recent call last):
        File "/httpstan/setup.py", line 2, in <module>
          from setuptools import setup
      ModuleNotFoundError: No module named 'setuptools'
      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-pit6zccr/overlay/lib/python3.10/site-packages/poetry/core/masonry/api.py", line 68, in build_wheel
          return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
        File "/tmp/pip-build-env-pit6zccr/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 78, in make_in
          wb.build()
        File "/tmp/pip-build-env-pit6zccr/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 110, in build
          self._build(zip_file)
        File "/tmp/pip-build-env-pit6zccr/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 162, in _build
          self._run_build_command(setup)
        File "/tmp/pip-build-env-pit6zccr/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 190, in _run_build_command
          subprocess.check_call(
        File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/usr/local/bin/python', '/httpstan/setup.py', 'build', '-b', '/httpstan/build']' returned non-zero exit status 1.

Attempt 2

Attempt to build package from source python setup.py sdist bdist_wheel resulted in a missing header by the looks of it

Attempt 2 Full Trace

root@87f1f8c9cb33:/httpstan# python setup.py sdist bdist_wheel
running sdist
running egg_info
creating httpstan.egg-info
writing httpstan.egg-info/PKG-INFO
writing dependency_links to httpstan.egg-info/dependency_links.txt
writing requirements to httpstan.egg-info/requires.txt
writing top-level names to httpstan.egg-info/top_level.txt
writing manifest file 'httpstan.egg-info/SOURCES.txt'
reading manifest file 'httpstan.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'httpstan.egg-info/SOURCES.txt'
running check
creating httpstan-4.7.2
creating httpstan-4.7.2/httpstan
creating httpstan-4.7.2/httpstan.egg-info
creating httpstan-4.7.2/httpstan/services
copying files to httpstan-4.7.2...
copying LICENSE -> httpstan-4.7.2
copying README.rst -> httpstan-4.7.2
copying pyproject.toml -> httpstan-4.7.2
copying setup.cfg -> httpstan-4.7.2
copying setup.py -> httpstan-4.7.2
copying httpstan/__init__.py -> httpstan-4.7.2/httpstan
copying httpstan/__main__.py -> httpstan-4.7.2/httpstan
copying httpstan/app.py -> httpstan-4.7.2/httpstan
copying httpstan/build_ext.py -> httpstan-4.7.2/httpstan
copying httpstan/cache.py -> httpstan-4.7.2/httpstan
copying httpstan/compile.py -> httpstan-4.7.2/httpstan
copying httpstan/config.py -> httpstan-4.7.2/httpstan
copying httpstan/empty.cpp -> httpstan-4.7.2/httpstan
copying httpstan/fits.py -> httpstan-4.7.2/httpstan
copying httpstan/models.py -> httpstan-4.7.2/httpstan
copying httpstan/openapi.py -> httpstan-4.7.2/httpstan
copying httpstan/routes.py -> httpstan-4.7.2/httpstan
copying httpstan/schemas.py -> httpstan-4.7.2/httpstan
copying httpstan/services_stub.py -> httpstan-4.7.2/httpstan
copying httpstan/socket_logger.hpp -> httpstan-4.7.2/httpstan
copying httpstan/socket_writer.hpp -> httpstan-4.7.2/httpstan
copying httpstan/stan.pyi -> httpstan-4.7.2/httpstan
copying httpstan/stan_services.cpp -> httpstan-4.7.2/httpstan
copying httpstan/utils.py -> httpstan-4.7.2/httpstan
copying httpstan/views.py -> httpstan-4.7.2/httpstan
copying httpstan.egg-info/PKG-INFO -> httpstan-4.7.2/httpstan.egg-info
copying httpstan.egg-info/SOURCES.txt -> httpstan-4.7.2/httpstan.egg-info
copying httpstan.egg-info/dependency_links.txt -> httpstan-4.7.2/httpstan.egg-info
copying httpstan.egg-info/requires.txt -> httpstan-4.7.2/httpstan.egg-info
copying httpstan.egg-info/top_level.txt -> httpstan-4.7.2/httpstan.egg-info
copying httpstan/services/__init__.py -> httpstan-4.7.2/httpstan/services
copying httpstan/services/arguments.py -> httpstan-4.7.2/httpstan/services
copying httpstan/services/cmdstan-help-all.json -> httpstan-4.7.2/httpstan/services
Writing httpstan-4.7.2/setup.cfg
creating dist
Creating tar archive
removing 'httpstan-4.7.2' (and everything under it)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/build_ext.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/compile.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/views.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/services_stub.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/openapi.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/__main__.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/__init__.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/fits.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/routes.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/config.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/schemas.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/app.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/utils.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/models.py -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/cache.py -> build/lib.linux-x86_64-cpython-310/httpstan
creating build/lib.linux-x86_64-cpython-310/httpstan/services
copying httpstan/services/arguments.py -> build/lib.linux-x86_64-cpython-310/httpstan/services
copying httpstan/services/__init__.py -> build/lib.linux-x86_64-cpython-310/httpstan/services
copying httpstan/stan_services.cpp -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/socket_writer.hpp -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/stan.pyi -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/empty.cpp -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/socket_logger.hpp -> build/lib.linux-x86_64-cpython-310/httpstan
copying httpstan/services/cmdstan-help-all.json -> build/lib.linux-x86_64-cpython-310/httpstan/services
running build_ext
building 'httpstan.empty' extension
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/httpstan
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ihttpstan/include -I/usr/local/include/python3.10 -c httpstan/empty.cpp -o build/temp.linux-x86_64-cpython-310/httpstan/empty.o -fvisibility=hidden -std=c++14
httpstan/empty.cpp:1:10: fatal error: pybind11/pybind11.h: No such file or directory
    1 | #include <pybind11/pybind11.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

I just tried pystan==3.0.0 which seemed to install OK, if that helps.

ahartikainen commented 2 years ago

To install httpstan from source see the following instructions

https://httpstan.readthedocs.io/en/latest/installation.html

samhiggs commented 2 years ago

OK so did that, and got the a similar result.

root@5938bd87209c:/httpstan# python3 -m poetry build
Creating virtualenv httpstan-tJ3t_SzP-py3.9 in /root/.cache/pypoetry/virtualenvs
Building httpstan (4.7.2)
  - Building sdist
  - Built httpstan-4.7.2.tar.gz
  - Building wheel
running build
running build_py
creating /httpstan/build
creating /httpstan/build/lib.linux-x86_64-cpython-39
creating /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/build_ext.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/compile.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/views.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/services_stub.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/openapi.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/__main__.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/__init__.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/fits.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/routes.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/config.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/schemas.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/app.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/utils.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/models.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/cache.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
creating /httpstan/build/lib.linux-x86_64-cpython-39/httpstan/services
copying httpstan/services/arguments.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan/services
copying httpstan/services/__init__.py -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan/services
copying httpstan/stan_services.cpp -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/socket_writer.hpp -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/stan.pyi -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/empty.cpp -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/socket_logger.hpp -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan
copying httpstan/services/cmdstan-help-all.json -> /httpstan/build/lib.linux-x86_64-cpython-39/httpstan/services
running build_ext
building 'httpstan.empty' extension
creating /httpstan/build/temp.linux-x86_64-cpython-39
creating /httpstan/build/temp.linux-x86_64-cpython-39/httpstan
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ihttpstan/include -I/root/.cache/pypoetry/virtualenvs/httpstan-tJ3t_SzP-py3.9/include -I/usr/local/include/python3.9 -c httpstan/empty.cpp -o /httpstan/build/temp.linux-x86_64-cpython-39/httpstan/empty.o -fvisibility=hidden -std=c++14
httpstan/empty.cpp:1:10: fatal error: pybind11/pybind11.h: No such file or directory
    1 | #include <pybind11/pybind11.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

  CalledProcessError

  Command '['/root/.cache/pypoetry/virtualenvs/httpstan-tJ3t_SzP-py3.9/bin/python', '/httpstan/setup.py', 'build', '-b', '/httpstan/build']' returned non-zero exit status 1.

  at /usr/local/lib/python3.9/subprocess.py:373 in check_call
       369│     if retcode:
       370│         cmd = kwargs.get("args")
       371│         if cmd is None:
       372│             cmd = popenargs[0]
    →  373│         raise CalledProcessError(retcode, cmd)
       374│     return 0
       375│
       376│
       377│ def check_output(*popenargs, timeout=None, **kwargs):
ahartikainen commented 2 years ago

https://github.com/stan-dev/httpstan/blob/main/.github/workflows/main.yml

Check how httpstan does it in workflows.

riddell-stan commented 2 years ago

Judging by https://github.com/stan-dev/pystan/issues/350#issuecomment-1164047384, it looks like @samhiggs is having trouble installing httpstan from source. (Is make being run, as indicated in the linked httpstan installation docs?)

Since this isn't a bug report, I'm going to close it. I think the right place for this is the Stan forums.