search

aicoe-aiops / mailing-list-analysis-toolkit

Opensource mailing list analysis with open data hub
Other
0 stars 7 forks source link

pipenv install fails #68

Closed oindrillac closed 3 years ago

oindrillac commented 3 years ago

Describe the bug Pipenv install of hatesonar and scikit-learn==0.20.3 fails.

To Reproduce

Trying out similar workflow from https://github.com/aicoe-aiops/mailing-list-analysis-toolkit/issues/51#issuecomment-781634183 fails.

cc: @cdolfi related to : #54

MichaelClifford commented 3 years ago

@oindrillac why are you forcing scikit-learn==0.20.3 I'm trying to run this on the s2i-elyra-v6 image and it looks like both the language packages, hatesonar and flair require newer versions of sk-learn

MichaelClifford commented 3 years ago

By not specifying the versions for hatesonar or scikit-learn in the Pipfile, running pipenv lock --pre then pipenv install seems to work.

oindrillac commented 3 years ago

@MichaelClifford hatesonar with a scikit-learn > 0.20.3 was giving this error

The problem above is not only with installing scikit-learn==0.20.3.

Noticed the same when I tried

pipenv install --pre hatesonar as well on top of the Pipfile on upstream.

I'm trying to run this on the s2i-elyra-v6 image

Doesnt the elyra v6 image support 3.8? The repo is on version python 3.6, and hence I've been trying this on the mailing-list image.

oindrillac commented 3 years ago

@MichaelClifford on a version greater than 0.20.3

sonar = Sonar() gives the following error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-b474975b2fe6> in <module>
----> 1 sonar = Sonar()

/opt/app-root/lib/python3.6/site-packages/hatesonar/api.py in __init__(self)
     19         model_file = os.path.join(BASE_DIR, 'model.joblib')
     20         preprocessor_file = os.path.join(BASE_DIR, 'preprocess.joblib')
---> 21         self.estimator = joblib.load(model_file)
     22         self.preprocessor = joblib.load(preprocessor_file)
     23 

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in load(filename, mmap_mode)
    583                     return load_compatibility(fobj)
    584 
--> 585                 obj = _unpickle(fobj, filename, mmap_mode)
    586     return obj

/opt/app-root/lib/python3.6/site-packages/joblib/numpy_pickle.py in _unpickle(fobj, filename, mmap_mode)
    502     obj = None
    503     try:
--> 504         obj = unpickler.load()
    505         if unpickler.compat_mode:
    506             warnings.warn("The file '%s' has been generated with a "

/usr/lib64/python3.6/pickle.py in load(self)
   1048                     raise EOFError
   1049                 assert isinstance(key, bytes_types)
-> 1050                 dispatch[key[0]](self)
   1051         except _Stop as stopinst:
   1052             return stopinst.value

/usr/lib64/python3.6/pickle.py in load_global(self)
   1336         module = self.readline()[:-1].decode("utf-8")
   1337         name = self.readline()[:-1].decode("utf-8")
-> 1338         klass = self.find_class(module, name)
   1339         self.append(klass)
   1340     dispatch[GLOBAL[0]] = load_global

/usr/lib64/python3.6/pickle.py in find_class(self, module, name)
   1386             elif module in _compat_pickle.IMPORT_MAPPING:
   1387                 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1388         __import__(module, level=0)
   1389         if self.proto >= 4:
   1390             return _getattribute(sys.modules[module], name)[0]

ModuleNotFoundError: No module named 'sklearn.linear_model.logistic'
oindrillac commented 3 years ago

since the hatesonar approach is experimental and is most likely not the final approach, is it a good idea not to track the dependencies in the pipfile and lock

chauhankaranraj commented 3 years ago 
ERROR: ERROR: Could not find a version that matches black
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1,18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies.

@oindrillac From the error message, it looks like the issue is not hatesonar or sklearn, but rather black. What happens when you explicitly mention black version in the pipfile like this:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

The above works fine for me on operate-first JH, on the mailing-list-analysis-toolkit image.

oindrillac commented 3 years ago

@chauhankaranraj thanks for the tip

black seems to stop throwing an error after this.

however, i think the scikit-learn version is still throws a conflict when flair is added

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipython = "*"

[packages]
pandas = "*"
boto3 = "*"
wget = "*"
python-dotenv = "*"
sklearn = "*"
papermill = "*"
ibm-cloud-sdk-core = "*"
ibm-watson = "*"
flair = "*"
scikit-learn = "==0.20.3"
hatesonar = "*"
black = "==20.8b1"

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = false

[thoth.allow_prereleases]
black = true

getting the version mismatch issue

✘ Locking Failed! 
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/app-root/lib64/python3.8/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches scikit-learn==0.20.3,>=0.19.1,>=0.21.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
Tried: 0.9, 0.10, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.14, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.17, 0.17.1, 0.18, 0.18.1, 0.18.2, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.21.1, 0.21.2, 0.21.3, 0.22, 0.22, 0.22.1, 0.22.1, 0.22.2, 0.22.2.post1, 0.22.2.post1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.23.2, 0.23.2, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.24.1
Skipped pre-versions: 0.15.0b1, 0.15.0b2, 0.16b1, 0.17b1, 0.19b2, 0.20rc1, 0.21rc2, 0.22rc2.post1, 0.22rc3, 0.22rc3, 0.23.0rc1, 0.23.0rc1, 0.24.dev0, 0.24.dev0, 0.24.0rc1, 0.24.0rc1
There are incompatible versions in the resolved dependencies:
  scikit-learn==0.20.3 (from -r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 4))
  scikit-learn (from sklearn==0.0->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 8))
  scikit-learn>=0.19.1 (from hatesonar==0.0.7->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 9))
  scikit-learn>=0.21.3 (from flair==0.8.0.post1->-r /tmp/pipenvt5kai5arrequirements/pipenv-2n35nu3h-constraints.txt (line 3))

and I am not sure how we can overcome this other than not tracking the scikit-learn pinned dependency in the Pipfile.

MichaelClifford commented 3 years ago

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?

pacospace commented 3 years ago

@pacospace do you think we could mitigate this issue by using "overlays" and having notebook specific dependencies? If so, we should look into using those here. This is a good first use case for a Elyra/KFpipeline that actually requires unique runtime images for individual notebooks. How does the aicoe-ci/build-check handle repo's with out a global Pipfile?

Thanks @MichaelClifford, I agree could be a very nice use case.

The tutorial we have shows how the repo will look like: https://github.com/thoth-station/elyra-aidevsecops-tutorial

Basically each notebook will have its own kernel and overlays/ repo matches kernel name selected for the notebook. If you use jupyterlab-requirements all kernels and overlays directory will be updated automatically. And the dependecies and kernel name will be present in notebook metadata as well.

So steps are:

Then when you open a release the aicoe-ci pipeline will do the magic and each overlays will have an image on your registry depending on the name you specify in the .aicoe.yaml. Then your images are ready to be used in the Elyra Pipeline.