ValueError: unsupported pickle protocol: 5 - Numpy 1.13.3 or above is required for this version of scipy (detected version 1.11.3)

[Y-valencia]

Hello, I've been trying this for days now, so i come to ask for help. When installing the regular version of lcopt-dev,I got the version 0.4.2, so I try to force the installation as said in #72 but got this error: -the version of numpy needs to be 1.13.3 or above = when trying to update i get a cuda driver issue -i make this env in python 3.6, i also tried to run the env in python 3.8 and didn't work either

does anyone have the same problem? or some recommendations about which path should i follow to succesfully install it?

thanks in advance

The error i get:

C:\Users\y\anaconda3\envs\lcopt-dev\lib\site-packages\scipy__init__.py:115: UserWarning: Numpy 1.13.3 or above is required for this version of scipy (detected version 1.11.3) UserWarning)

ValueError Traceback (most recent call last)

in ----> 1 from lcopt import * ~\anaconda3\envs\lcopt-dev\lib\site-packages\lcopt\__init__.py in 3 """ 4 from lcopt.io import * ----> 5 from lcopt.model import * 6 from lcopt.export_view import * 7 from lcopt.bw2_import import * ~\anaconda3\envs\lcopt-dev\lib\site-packages\lcopt\model.py in 9 from lcopt.io import * 10 #from lcopt.ipython_interactive import IFS ---> 11 from lcopt.interact import FlaskSandbox 12 from lcopt.bw2_export import Bw2Exporter 13 from lcopt.analysis import Bw2Analysis ~\anaconda3\envs\lcopt-dev\lib\site-packages\lcopt\interact.py in 12 from lcopt.bw2_export import Bw2Exporter 13 from lcopt.export_view import LcoptView ---> 14 from lcopt.utils import find_port 15 from lcopt.settings import settings 16 ~\anaconda3\envs\lcopt-dev\lib\site-packages\lcopt\utils.py in 12 import io 13 import os ---> 14 import eidl 15 from functools import partial 16 import yaml ~\anaconda3\envs\lcopt-dev\lib\site-packages\eidl\__init__.py in ----> 1 from .core import EcoinventDownloader, get_ecoinvent 2 from .storage import eidlstorage 3 4 __all__ = ['EcoinventDownloader', 'get_ecoinvent', 'eidlstorage'] ~\anaconda3\envs\lcopt-dev\lib\site-packages\eidl\core.py in 7 import requests 8 import bs4 ----> 9 import brightway2 as bw 10 11 from eidl.storage import eidlstorage ~\anaconda3\envs\lcopt-dev\lib\site-packages\brightway2\__init__.py in 1 # -*- coding: utf-8 -* ----> 2 from bw2data import * 3 from bw2calc import * 4 from bw2io import * 5 ~\anaconda3\envs\lcopt-dev\lib\site-packages\bw2data\__init__.py in 33 from .project import projects 34 from .utils import set_data_dir ---> 35 from .meta import ( 36 dynamic_calculation_setups, 37 calculation_setups, ~\anaconda3\envs\lcopt-dev\lib\site-packages\bw2data\meta.py in 190 preferences = Preferences() 191 weightings = WeightingMeta() --> 192 calculation_setups = CalculationSetups() 193 dynamic_calculation_setups = DynamicCalculationSetups() ~\anaconda3\envs\lcopt-dev\lib\site-packages\bw2data\serialization.py in __init__(self, dirpath) 120 self.filename 121 ) --> 122 self.load() 123 124 def load(self): ~\anaconda3\envs\lcopt-dev\lib\site-packages\bw2data\serialization.py in load(self) 125 """Load the serialized data. Creates the file if not yet present.""" 126 try: --> 127 self.data = self.deserialize() 128 except IOError: 129 # Create if not present ~\anaconda3\envs\lcopt-dev\lib\site-packages\bw2data\serialization.py in deserialize(self) 238 239 def deserialize(self): --> 240 return self.unpack(pickle.load(open(self.filepath, "rb"))) 241 242 **ValueError: unsupported pickle protocol: 5**

[pjamesjoyce]

Hi - The root of this problem (ValueError: unsupported pickle protocol: 5) comes from an issue with brightway2-data, which should have been resolved by the commit they pushed in mid-April (https://github.com/brightway-lca/brightway2-data/commit/09f13b22e651e0d496f0eff63ef984c4e8ea637d)

Can you try updating brightway (or just the brightway2-data package) and see if that helps?

The issue (described here https://github.com/brightway-lca/brightway2-data/issues/68) is that brightway2data used to use the highest available pickle protocol to save its data files. Python 3.8 introduced pickle 5 which means that if you ever open something in a python 3.8 brightway environment, it breaks backwards compatability to any 3.6/3.7 environments as they don't have the right version of pickle to open any data files. The commit above hard codes pickle 4 as the protocol to use in order to try and restore backwards compatability.

You might need to open up your 3.8 environment, load up brightway, load up the LCOPT_SETUP projects within brightway (using projects to get a list and projects.set_current('project name') to open them up) in order to re-save the datafiles in a version the lcopt environment can understand. Does that make sense?

I'll look into what's going on with numpy/scipy versions - there's some kind of conflict in the conda environments - but this should work as a short term fix.

Let me know if this helps and I can close this issue.

[pjamesjoyce]

Just checked and the latest pypi distribution of bw2data doesn't include the update, so updating via pip won't help.

[pjamesjoyce]

The conda distribution doesn't include the new bugfix either

[pjamesjoyce]

My guess is that it's coming as part of a bigger update - installing the dev version of just bw2data from github breaks brightway - I'll work on getting lcopt python 3.8 compatible and get back to you

[pjamesjoyce]

ok - this is a highly unrecommended fix, messing with the code in existing packages in your environment can cause unintended problems

Having said that it worked for me and doesn't seem to have had any other ill effects yet...

1. Find where bw2data is installed for your python 3.8 environment - most likely something like C:\Users\<username>\AppData\Local\Continuum\miniconda3\envs\<env name>\Lib\site-packages\bw2data
2. Use something like Sublime text text editor to find pickle.HIGHEST_PROTOCOL in all files and replace it with 4 (it should find 6 instances in 5 files - data_store.py, serialization.py, sqlite.py, base.py and database.py)
3. Open up a jupyter notebook or python shell and run the following:

from brightway2 import *
import os
from bw2data.serialization import PickledDict

# Subclass PickledDict to give it a filename (but nothing else)
class RePickler(PickledDict):
    def __init__(self, filename, *args, **kwargs):
        self.filename = filename
        super(RePickler, self).__init__(*args, **kwargs)

# Find all of the pickles that are currently in protocol 5
pickle_list = [x for x in os.listdir(projects.dir) if os.path.splitext(x)[1] == '.pickle']

# Repickle them in protocol 4
for x in pickle_list:
    repickler = RePickler(x)
    repickler.serialize()
    repickler = None