Docker problem: ValueError: numpy.ufunc has the wrong size, try recompiling.

[rcoenen]

when trying to start japonicus (using Docker) I get this error:

ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

full copy+paste from my terminal

$ docker run --rm japonicus
Traceback (most recent call last):
  File "/opt/japonicus/japonicus-run", line 4, in <module>
    import japonicus
  File "/opt/japonicus/japonicus/__init__.py", line 3, in <module>
    from .japonicus import *
  File "/opt/japonicus/japonicus/japonicus.py", line 9, in <module>
    from .evolution_generations import gekko_generations
  File "/opt/japonicus/japonicus/evolution_generations.py", line 8, in <module>
    import promoterz
  File "/opt/japonicus/promoterz/__init__.py", line 11, in <module>
    from .import webServer
  File "/opt/japonicus/promoterz/webServer/__init__.py", line 3, in <module>
    from .import core
  File "/opt/japonicus/promoterz/webServer/core.py", line 7, in <module>
    import dash
  File "/usr/local/lib/python3.6/site-packages/dash/__init__.py", line 1, in <module>
    from .dash import Dash  # noqa: F401
  File "/usr/local/lib/python3.6/site-packages/dash/dash.py", line 23, in <module>
    import plotly
  File "/usr/local/lib/python3.6/site-packages/plotly/__init__.py", line 31, in <module>
    from plotly import (plotly, dashboard_objs, graph_objs, grid_objs, tools,
  File "/usr/local/lib/python3.6/site-packages/plotly/plotly/__init__.py", line 10, in <module>
    from . plotly import (
  File "/usr/local/lib/python3.6/site-packages/plotly/plotly/plotly.py", line 32, in <module>
    from _plotly_utils.basevalidators import CompoundValidator, is_array
  File "/usr/local/lib/python3.6/site-packages/_plotly_utils/basevalidators.py", line 25, in <module>
    pd = import_module('pandas')
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 25, in <module>
    from pandas import hashtable, tslib, lib
  File "pandas/src/numpy.pxd", line 865, in init pandas.hashtable (pandas/hashtable.c:38368)
ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

[rcoenen]

also getting some incompatibility issues

dash 0.39.0 has requirement dash-core-components==0.44.0, but you'll have dash-core-components 0.22.1 which is incompatible.
dash 0.39.0 has requirement dash-html-components==0.14.0, but you'll have dash-html-components 0.10.1 which is incompatible.
dash 0.39.0 has requirement dash-renderer==0.20.0, but you'll have dash-renderer 0.21.0 which is incompatible.

[rcoenen]

answering own question, partly:

in Japonicus' dir requirements.txt change these lines

from numpy==1.14.5 to numpy==1.16.2 from dash_core_components==0.22.1 to dash_core_components==0.44.0 from dash_html_components==0.10.1 to dash_html_components==0.14.0 from dash_renderer to dash_renderer==0.20.0 (without a version it defaulted to 0.21.0 which was incompatible)

With these changes I can build and run Japnicus: docker run --rm japonicus

[rcoenen]

Next problem after running docker run --rm japonicus

It now throws

Error importing optional module pandas
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/plotly/optional_imports.py", line 25, in get_module
    return import_module(name)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 39, in <module>
    from pandas.core.api import *
  File "/usr/local/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
    from pandas.core.groupby import Grouper
  File "/usr/local/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.groupby import GroupBy  # noqa: F401
  File "/usr/local/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 20, in <module>
    import pandas.compat as compat
AttributeError: module 'pandas' has no attribute 'compat'
Usage: japonicus-run [options]

[rcoenen]

Despite the pandas error above, Japonicus seems to be working. It at least shows this:

Usage: japonicus-run [options]

Options:
  -h, --help            show this help message and exit
  -g, --genetic         Genetic Algorithm evolution mode.
  -c, --chromosome      Alternative internal representation of parameters for
                        Genetic Algorithm mode.
  -b, --bayesian        Bayesian evolution mode.
  -k, --gekko           Launch gekko instance.
  -r, --random          Run on random strategy.
  -e, --benchmark       Run GA benchmark mode. Strategy names are restricted
                        to specific strats.
  -w, --web             Launch japonicus web server showing evolutionary
                        statistics.
  --repeat <x>=REPEATER
  --strat <strat>=STRATEGY
  --skeleton <skeleton>=SKELETON
  --parameter_spread=PARAMETER_SPREAD
  --POP_SIZE=POP_SIZE   
  --NBEPOCH=NBEPOCH     
  --NBLOCALE=NBLOCALE   
  --cxpb=CXPB           
  --mutpb=MUTPB         
  --_lambda=_LAMBDA     
  --PRoFIGA_beta=PROFIGA_BETA
  --localeExpirationAge=LOCALEEXPIRATIONAGE
  --localeExplodeChance=LOCALEEXPLODECHANCE
  --localeCreationChance=LOCALECREATIONCHANCE
  --localeWalkChance=LOCALEWALKCHANCE
  --localeWalkDistance=LOCALEWALKDISTANCE
  --worldSeedSize=WORLDSEEDSIZE
  --minimumProfitFilter=MINIMUMPROFITFILTER
  --chromosome.GeneSize=CHROMOSOME.GENESIZE
  --chromosome.Density=CHROMOSOME.DENSITY
  --weights.profit=WEIGHTS.PROFIT
  --weights.sharpe=WEIGHTS.SHARPE
  --dataset_span=DATASET_SPAN
  --eval_dataset_span=EVAL_DATASET_SPAN
  --dataset_source.exchange=DATASET_SOURCE.EXCHANGE
  --dataset_source.currency=DATASET_SOURCE.CURRENCY
  --dataset_source.asset=DATASET_SOURCE.ASSET
  --dataset_source2.exchange=DATASET_SOURCE2.EXCHANGE
  --dataset_source2.currency=DATASET_SOURCE2.CURRENCY
  --dataset_source2.asset=DATASET_SOURCE2.ASSET
  --eval_dataset_source.exchange=EVAL_DATASET_SOURCE.EXCHANGE
  --eval_dataset_source.currency=EVAL_DATASET_SOURCE.CURRENCY
  --eval_dataset_source.asset=EVAL_DATASET_SOURCE.ASSET
  --gekkoDebug=GEKKODEBUG
  --deltaDays=DELTADAYS
  --candleSize=CANDLESIZE
  --interpreteBacktestProfit=INTERPRETEBACKTESTPROFIT
  --ParallelCandlestickDataset=PARALLELCANDLESTICKDATASET
  --ParallelBacktests=PARALLELBACKTESTS
  --NBBESTINDS=NBBESTINDS
  --NBADDITIONALINDS=NBADDITIONALINDS
  --evaluateSettingsPeriodically=EVALUATESETTINGSPERIODICALLY
  --proofSize=PROOFSIZE

[Gab0]

Hey, this last message is it running inside docker (composed with gekko)? Then it is working. Ok, I'm updating to this latest numpy, and dash also. This additional dash stuff like core-components is now installed automatically by dash (thank god) so just removed from requirements.txt.

The pandas error has no side effects I think, it is optional for dash. Pandas version is likely to be unmatched with numpy's at this point. I think japonicus does not use pandas, dodged it just to avoid complications on docker...

[Gab0]

Also, did you also try the docker-compose file at the repository, not from wiki?

[rcoenen]

I copied everything from the wiki... this is my very first attempt ever at using Docker so.. mileage might vary. I'll try again with the docker-compose file from the repository. (Im not even sure what that means but will figure it out)

[Gab0]

I mean this https://github.com/Gab0/japonicus/blob/master/docker-compose.yaml