Availability on Conda Repo

[calvinwy]

Is it possible to make Gekko available on Conda repository (eg. conda-forge)? I am using Gekko in one of my package, and I wanted to include a environment.yml for default packages dependency for conda environment.

[calvinwy]

Also, I cannot publish my package on conda-forge due to the its dependency on Gekko, since Gekko is not yet on conda-forge. Could someone please kindly look into this... Thanks a lot!

https://github.com/conda-forge/staged-recipes/runs/6348196472

[APMonitor]

There are no immediate plans to publish Gekko on conda-forge.

YAML can handle both capturing and reinstalling Pip-installed packages with:

conda env export > environment.yaml

you can recreate with pip dependencies on another platform:

conda env create -n m1 -f environment.yaml

Most eveything that is on conda is also on pypi (pip), in addition to many more packages that are exclusively on pypi. Have you considered switching over to pypi for your package distribution? Once it is in pypi, it appears possible to build a conda-forge package (see conda skeleton and building conda packages).

I added this to the list of future enhancements.

[APMonitor]

Tried the conda skeleton pypi gekko route to build a conda-forge package:

(base) C:\conda>conda skeleton pypi gekko
Warning, the following versions were found for gekko
0.0.1rc1
0.0.2a1
0.0.2a2
0.0.2b1
0.0.2b2
0.0.2rc1
0.0.2rc2
0.0.2rc3
0.0.2rc4
0.0.3a1
0.0.3a2
0.0.3a3
0.0.3b1
0.0.4a1
0.0.4a2
0.0.4a3
0.0.4a4
0.0.4a5
0.0.4a6
0.0.4a7
0.0.4a8
0.0.4a9
0.0.4b1
0.0.4b2
0.0.4rc1
0.0.4rc2
0.0.4rc3
0.1a1
0.1a2
0.1a3
0.1b1
0.1b2
0.1b3
0.1rc1
0.1rc2
0.1rc3
0.1rc7
0.1rc8
0.1rc9
0.1
0.2b1
0.2b2
0.2rc4
0.2rc5
0.2rc6
0.2.0
0.2.1
0.2.2
0.2.3
0.2.4
0.2.5
0.2.6
0.2.7
0.2.8
1.0.0
1.0.1
1.0.2
Using 1.0.2
Use --version to specify a different version.
Using url https://files.pythonhosted.org/packages/b0/c0/72e179f66b159c0f441b7961e74c73e06fd14a90eb4952eafad686446b95/gekko-1.0.2.tar.gz (11.7 MB) for gekko.
Downloading gekko
PyPI URL:  https://files.pythonhosted.org/packages/b0/c0/72e179f66b159c0f441b7961e74c73e06fd14a90eb4952eafad686446b95/gekko-1.0.2.tar.gz
Using cached download
Unpacking gekko...
done
working in C:\Users\johnh\AppData\Local\Temp\tmpz_p84wilconda_skeleton_gekko-1.0.2.tar.gz
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env

The following NEW packages will be INSTALLED:

    ca-certificates:   2022.4.26-haa95532_0
    certifi:           2021.10.8-py38haa95532_2
    m2-gcc-libs:       5.3.0-4
    m2-msys2-runtime:  2.5.0.17080.65c939c-3
    m2-patch:          2.7.5-2
    msys2-conda-epoch: 20160418-1
    openssl:           1.1.1o-h2bbff1b_0
    pip:               21.2.2-py38haa95532_0
    python:            3.8.13-h6244533_0
    pyyaml:            6.0-py38h2bbff1b_1
    setuptools:        61.2.0-py38haa95532_0
    sqlite:            3.38.3-h2bbff1b_0
    vc:                14.2-h21ff451_1
    vs2015_runtime:    14.27.29016-h5e58377_2
    wheel:             0.37.1-pyhd3eb1b0_0
    wincertstore:      0.2-py38haa95532_2
    yaml:              0.2.5-he774522_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Applying patch: C:\Users\johnh\AppData\Local\Temp\tmpz_p84wilconda_skeleton_gekko-1.0.2.tar.gz\pypi-distutils.patch
Applying patch: C:\Users\johnh\AppData\Local\Temp\tmpz_p84wilconda_skeleton_gekko-1.0.2.tar.gz\pypi-distutils.patch with args:
['-p0', '-i', 'C:\\Users\\johnh\\AppData\\Local\\Temp\\tmp5z1ofsjd\\pypi-distutils.patch.native', '--binary']
checking file core.py
Hunk #1 succeeded at 167 with fuzz 2 (offset 1 line).
patching file core.py
Hunk #1 succeeded at 167 with fuzz 2 (offset 1 line).
running install
C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing gekko.egg-info\PKG-INFO
writing dependency_links to gekko.egg-info\dependency_links.txt
writing requirements to gekko.egg-info\requires.txt
writing top-level names to gekko.egg-info\top_level.txt
reading manifest file 'gekko.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gekko.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build
creating build\lib
creating build\lib\gekko
copying gekko\apm.py -> build\lib\gekko
copying gekko\brain.py -> build\lib\gekko
copying gekko\chemical.py -> build\lib\gekko
copying gekko\gekko.py -> build\lib\gekko
copying gekko\gk_debug.py -> build\lib\gekko
copying gekko\gk_global_options.py -> build\lib\gekko
copying gekko\gk_gui.py -> build\lib\gekko
copying gekko\gk_operators.py -> build\lib\gekko
copying gekko\gk_parameter.py -> build\lib\gekko
copying gekko\gk_post_solve.py -> build\lib\gekko
copying gekko\gk_variable.py -> build\lib\gekko
copying gekko\gk_write_files.py -> build\lib\gekko
copying gekko\properties.py -> build\lib\gekko
copying gekko\__init__.py -> build\lib\gekko
creating build\lib\gekko\static
copying gekko\static\index.html -> build\lib\gekko\static
creating build\lib\gekko\static\static
copying gekko\static\static\bootstrap.min.css -> build\lib\gekko\static\static
copying gekko\static\static\GekkoLogo.png -> build\lib\gekko\static\static
creating build\lib\gekko\static\static\css
copying gekko\static\static\css\app.c68ba9dcf7bd08c3aad2e946c455bc61.css -> build\lib\gekko\static\static\css
copying gekko\static\static\css\app.c68ba9dcf7bd08c3aad2e946c455bc61.css.map -> build\lib\gekko\static\static\css
creating build\lib\gekko\static\static\js
copying gekko\static\static\js\app.943e6331e1599b12b7df.js -> build\lib\gekko\static\static\js
copying gekko\static\static\js\app.943e6331e1599b12b7df.js.map -> build\lib\gekko\static\static\js
copying gekko\static\static\js\manifest.13b220bf0bb1548816a4.js -> build\lib\gekko\static\static\js
copying gekko\static\static\js\manifest.13b220bf0bb1548816a4.js.map -> build\lib\gekko\static\static\js
copying gekko\static\static\js\vendor.0ec4a591b473bfdae464.js -> build\lib\gekko\static\static\js
copying gekko\static\static\js\vendor.0ec4a591b473bfdae464.js.map -> build\lib\gekko\static\static\js
creating build\lib\gekko\bin
copying gekko\bin\apm.exe -> build\lib\gekko\bin
copying gekko\bin\apm -> build\lib\gekko\bin
copying gekko\bin\apm_arm -> build\lib\gekko\bin
copying gekko\bin\apm_mac -> build\lib\gekko\bin
creating build\bdist.win-amd64
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\apm.py -> build\bdist.win-amd64\egg\gekko
creating build\bdist.win-amd64\egg\gekko\bin
copying build\lib\gekko\bin\apm -> build\bdist.win-amd64\egg\gekko\bin
copying build\lib\gekko\bin\apm.exe -> build\bdist.win-amd64\egg\gekko\bin
copying build\lib\gekko\bin\apm_arm -> build\bdist.win-amd64\egg\gekko\bin
copying build\lib\gekko\bin\apm_mac -> build\bdist.win-amd64\egg\gekko\bin
copying build\lib\gekko\brain.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\chemical.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gekko.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_debug.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_global_options.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_gui.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_operators.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_parameter.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_post_solve.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_variable.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\gk_write_files.py -> build\bdist.win-amd64\egg\gekko
copying build\lib\gekko\properties.py -> build\bdist.win-amd64\egg\gekko
creating build\bdist.win-amd64\egg\gekko\static
copying build\lib\gekko\static\index.html -> build\bdist.win-amd64\egg\gekko\static
creating build\bdist.win-amd64\egg\gekko\static\static
copying build\lib\gekko\static\static\bootstrap.min.css -> build\bdist.win-amd64\egg\gekko\static\static
creating build\bdist.win-amd64\egg\gekko\static\static\css
copying build\lib\gekko\static\static\css\app.c68ba9dcf7bd08c3aad2e946c455bc61.css -> build\bdist.win-amd64\egg\gekko\static\static\css
copying build\lib\gekko\static\static\css\app.c68ba9dcf7bd08c3aad2e946c455bc61.css.map -> build\bdist.win-amd64\egg\gekko\static\static\css
copying build\lib\gekko\static\static\GekkoLogo.png -> build\bdist.win-amd64\egg\gekko\static\static
creating build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\app.943e6331e1599b12b7df.js -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\app.943e6331e1599b12b7df.js.map -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\manifest.13b220bf0bb1548816a4.js -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\manifest.13b220bf0bb1548816a4.js.map -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\vendor.0ec4a591b473bfdae464.js -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\static\static\js\vendor.0ec4a591b473bfdae464.js.map -> build\bdist.win-amd64\egg\gekko\static\static\js
copying build\lib\gekko\__init__.py -> build\bdist.win-amd64\egg\gekko
byte-compiling build\bdist.win-amd64\egg\gekko\apm.py to apm.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\brain.py to brain.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\chemical.py to chemical.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gekko.py to gekko.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_debug.py to gk_debug.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_global_options.py to gk_global_options.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_gui.py to gk_gui.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_operators.py to gk_operators.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_parameter.py to gk_parameter.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_post_solve.py to gk_post_solve.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_variable.py to gk_variable.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\gk_write_files.py to gk_write_files.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\properties.py to properties.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\gekko\__init__.py to __init__.cpython-38.pyc
creating build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\not-zip-safe -> build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\requires.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying gekko.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
creating dist
creating 'dist\gekko-1.0.2-py3.8.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing gekko-1.0.2-py3.8.egg
creating c:\users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\gekko-1.0.2-py3.8.egg
Extracting gekko-1.0.2-py3.8.egg to c:\users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages
Adding gekko 1.0.2 to easy-install.pth file

Installed c:\users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\gekko-1.0.2-py3.8.egg
Processing dependencies for gekko==1.0.2
Searching for numpy>=1.8
Reading https://pypi.org/simple/numpy/
C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\pkg_resources\__init__.py:122: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
Downloading https://files.pythonhosted.org/packages/fa/f2/f4ec28f935f980167740c5af5a1908090a48a564bed5e689f4b92386d7d9/numpy-1.22.3-cp38-cp38-win_amd64.whl#sha256=07a8c89a04997625236c5ecb7afe35a02af3896c8aa01890a849913a2309c676
Best match: numpy 1.22.3
Processing numpy-1.22.3-cp38-cp38-win_amd64.whl
Installing numpy-1.22.3-cp38-cp38-win_amd64.whl to c:\users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages
Adding numpy 1.22.3 to easy-install.pth file
Installing f2py-script.py script to C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\Scripts
Installing f2py.exe script to C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\Scripts

Installed c:\users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_h_env\lib\site-packages\numpy-1.22.3-py3.8-win-amd64.egg
Finished processing dependencies for gekko==1.0.2

Leaving build/test directories:
  Work:
 C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\work
  Test:
 C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\test_tmp
Leaving build/test environments:
  Test:
source activate  C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_test_env
  Build:
source activate  C:\Users\johnh\anaconda3\conda-bld\skeleton_1652221182185\_build_env

Traceback (most recent call last):
  File "C:\Users\johnh\anaconda3\Scripts\conda-skeleton-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\cli\main_skeleton.py", line 65, in main
    return execute(sys.argv[1:])
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\cli\main_skeleton.py", line 60, in execute
    api.skeletonize(args.packages, args.repo, output_dir=args.output_dir, recursive=args.recursive,
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\api.py", line 276, in skeletonize
    skeleton_return = module.skeletonize(packages, output_dir=output_dir, version=version,
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\skeletons\pypi.py", line 317, in skeletonize
    get_package_metadata(package, d, data, output_dir, python_version,
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\skeletons\pypi.py", line 731, in get_package_metadata
    requires = get_requirements(package, pkginfo, all_extras=all_extras)
  File "C:\Users\johnh\anaconda3\lib\site-packages\conda_build\skeletons\pypi.py", line 1097, in get_requirements
    for extra in pkginfo['extras_require']:
KeyError: 'extras_require'

A work-around is to use python3.6 or modify some of the skeleton\pypi.py file as shown here: https://github.com/conda/conda-build/issues/4354 It appears that this method for pypi to conda was built <=3.6 without recent updates to ensure compatibility.

[calvinwy]

@APMonitor try to use grayskull instead of skeleton to create the meta.yaml, it works for my package. (see the official step-by-step instruction guide here: https://conda-forge.org/docs/maintainer/adding_pkgs.html#step-by-step-instructions)

My experience with grayskull is that it creates a meta.yml instead of meta.yaml required by conda-forge, please remember to rename it. Second, the package version indicated in the meta.yaml file might have " around the version number, it would work once you erase them.

My package is already on pypi. The reason why I need to put this on conda-forge is because we use a cloud platform which the virtual machine instance is built based on conda and the jupyter notebook running on it is not working with packages installed using pip install. Also, having it on conda also make many data scientists' life easier.

Thanks a lot for these extra work.

[APMonitor]

grayskull created the meta.yaml file:

{% set name = "gekko" %}
{% set version = "1.0.2" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/gekko-{{ version }}.tar.gz
  sha256: c97034a58d25279b43f72386de43a62b41f901b14d6f2b5bdb3858ff746cd7d4

build:
  noarch: python
  script: {{ PYTHON }} -m pip install . -vv
  number: 0

requirements:
  host:
    - pip
    - python
  run:
    - numpy >=1.8
    - python

test:
  imports:
    - gekko
  commands:
    - pip check
  requires:
    - pip

about:
  home: https://github.com/BYU-PRISM/GEKKO
  summary: Machine learning and optimization for dynamic systems
  license: BSD-2-Clause
  license_file: LICENSE

extra:
  recipe-maintainers:
    - APMonitor

I submitted the forked staged-recipes for review with the gekko license and meta.yaml file in the examples folder. It is now up to the conda-forge maintainers to accept or reject the request. I appreciate the suggestion to use grayskull. It is much easier to submit than with skeleton.

https://github.com/APMonitor/staged-recipes

[calvinwy]

@APMonitor Thanks a lot for supporting! Once your is approve, I can submit mine too. Cheers!

[calvinwy]

@APMonitor by the way, I think you forgot to submit the pull request... you have to submit a pull request in order for them to accept it.

[APMonitor]

Thanks for the reminder. It looks like it is failing on some of the checks: https://github.com/conda-forge/staged-recipes/pull/18925 The error looks non-descriptive:

Check failure on line 1199 in Build log
@azure-pipelines azure-pipelines / staged-recipes (linux_64)
Build log #L1199
Bash exited with code '1'.

Any ideas?

[APMonitor]

Digging into the logs, it appears that the apm_arm local binary has a dependency that it doesn't like.

['  ERROR (gekko,lib/python3.10/site-packages/gekko/bin/apm_arm): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.\n.. is this binary repackaging?', '  ERROR (gekko,lib/python3.10/site-packages/gekko/bin/apm_arm): $RPATH/libc.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.\n.. is this binary repackaging?', '  ERROR (gekko,lib/python3.10/site-packages/gekko/bin/apm_arm): $RPATH/ld-linux-armhf.so.3 not found in packages, sysroot(s) nor the missing_dso_whitelist.\n.. is this binary repackaging?']
##[error]Bash exited with code '1'.

I need to update that binary (out of date).

[calvinwy]

I think something related to MacOS dependency, if you look at the error log, it says: Found Mach-O file but patching is only supported on macOS

The check passes on MacOS, but failed on Linux.

[calvinwy]

@APMonitor Perhaps, you can try to ping @conda-forge/staged-recipes @conda-forge/core on the PR if you cannot figure it out. I tried to look it up online, but no luck so far.

[APMonitor]

I released a new version of Gekko v1.0.4 and it resolved the issues with MacOS and Linux. There is an error on the Windows build.

INFO:conda_build.build:Packaging gekko
Packaging gekko
INFO:conda_build.build:Packaging gekko-1.0.4-pyhd8ed1ab_0
Packaging gekko-1.0.4-pyhd8ed1ab_0
number of files: 36
##[error]Cmd.exe exited with code '-1073741819'.
Finishing: Build recipe

Here is some information on that error code: https://docs.microsoft.com/en-us/answers/questions/379441/error-code-34exited-with-code-107374181934.html

The exit code -1073741819 in hex is 0xC0000005. If you look up this code in Microsoft's documentation (NTSTATUS Values) you will see that it indicates that your program was terminated due to an access violation. This error can occur for a variety of reasons, including de-referencing a NULL pointer or referencing an invalid address.

You should step through your code in the debugger line by line to determine where the access violation happens. That should enable you to analyze the situation and identify the cause.

Maybe getting closer...

[APMonitor]

Their documentation mentions that I should wait a few days before reaching out with a ping to their developer group. I tried setting python >=3.6 instead of python >=2.7 to see if it helps to limit the python version to 3+ but there is the same Windows error.

[APMonitor]

I just found this for Gekko 1.0.0: https://anaconda.org/Bjrn/gekko for conda install -c bjrn gekko. I haven't seen a notification of the conda-forge resolution of the error. Does this help you proceed with your package deployment with the gekko dependency?

[calvinwy]

I just found this for Gekko 1.0.0: https://anaconda.org/Bjrn/gekko for conda install -c bjrn gekko. I haven't seen a notification of the conda-forge resolution of the error. Does this help you proceed with your package deployment with the gekko dependency?

Unfortunately not, my package is required to be on conda-forge because of license issue (cannot put it on anaconda) and all dependencies are required to be on there as well. I think the conda-forge team just replied to you (see here), it seems like there is some precompiled component which makes the pull request fails... do u mind to have a look into that?

[APMonitor]

Thanks - I added a reply. The biggest problem seems to be with the apm binary. Perhaps we can remove the binaries from the conda-forge distribution. Users who need the local solve can always but them back into the bin folder with a wget command.

bin folder https://github.com/BYU-PRISM/GEKKO/tree/master/gekko/bin