Multiprocessing errors

[igt-misc]

Hello, I've been having a bit of an issue getting premise to run due to some sort of multiprocessing error.

I'm running python 3.9.13 and I've tried premise 1.6.3 and 1.6.9 on an ecoinvent 3.9.1 consequential database and keep getting the same issue. When I specify an existing brightway project and database, I see an inifinite series of "EXTRACTING", "IMPORTING", etc. messages along with the attached error message - this just keeps going forever too, even if I'm only importing a single scenario: premise_error.txt If I don't specify an existing brightway project, the error is slightly different, telling me it's a bw2io.error and that I should run it with use_mp=False (it still has the first error message in there, but now it includes the second one too).

Here's the code I'm using: python_that_fails.txt

I just copied it off of the jupyter notebooks, I've done minimal changes other than "putting in the decryption key" and "removing the decryption key when posting the code here".

Have I done something comically wrong here?

[romainsacchi]

Hi @igt-misc, your script looks OK. Do you confirm you're using premise 1.6.9? Is it possible to show me a larger excerpt of the error traceback?

[igt-misc]

I wish I could run it again to get the entire dump, but sadly I can't anymore - though I didn't actually fix it.

My computer was running Windows 10 when I ran the script, and knowing my colleague can run premise fine on Ubuntu I ran a smaller trial in a VM which worked fine - thus prompting me to reformat my machine to Ubuntu.

I can confirm however that I was running 1.6.9, and tried using both python 3.9 and 3.10.

This error might be related to my university's group policy on Windows - even running as administrator there were heavy restrictions on my read/write permissions in folders like AppData. My other hunch was that it had something to do with the overall thread count, as I've had concurrency issues in brightway before with read/write where it's told me to flag use_mp=False when I call writetodb, which struck me as odd seeing I have 8 cores/16 threads.

[romainsacchi]

OK. Well, let me know if you have an issue running it now.

[Stew-McD]

I have the same error only on consequential databases The cutoff dbs run fine.

Things I tried: fresh venvs and many different combinations of the following versions: Python==3.10/3.11 wurst==0.3.4 premise==1.6.9/1.7.0/1.7.1 bw2io==0.8.7/0.8.9/0.8.10

Seems like a wurst issue, from the traceback. But strange to only affect the consequential dbs, because I don't see any con specific files in the error message.

Maybe I'll try downgrading wurst.

Traceback:

/////////////////////// EXTRACTING IAM DATA ////////////////////////
Done!
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/stew/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/stew/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/stew/venvs/premise/lib/python3.10/site-packages/premise/electricity.py", line 198, in _update_electricity
    electricity.create_region_specific_power_plants()
  File "/home/stew/venvs/premise/lib/python3.10/site-packages/premise/electricity.py", line 1704, in create_region_specific_power_plants
    provider_ds = ws.get_one(
  File "/home/stew/venvs/premise/lib/python3.10/site-packages/wurst/searching.py", line 46, in get_one
    raise NoResults
wurst.errors.NoResults
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/stew/code/gh/WasteAndMaterialFootprint/src/WasteAndMaterialFootprint/FutureScenarios.py", line 157, in <module>
  File "/home/stew/code/gh/WasteAndMaterialFootprint/src/WasteAndMaterialFootprint/FutureScenarios.py", line 54, in wrapped
    return func(*args, **kwargs)
  File "/home/stew/code/gh/WasteAndMaterialFootprint/src/WasteAndMaterialFootprint/FutureScenarios.py", line 154, in make_premise_dbs
    # check databases
  File "/home/stew/venvs/premise/lib/python3.10/site-packages/premise/ecoinvent_modification.py", line 841, in update_electricity
    results = pool.starmap(_update_electricity, args)
  File "/home/stew/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/stew/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
wurst.errors.NoResults

[mkvdhulst]

I have a different issue, but since it is related to multiprocessing, I'll post it here, as it might be related to the same issue with wurst. I ran premise 1.7.0 for ecoinvent 3.9.1. cut-off, also by using the example jupyter notebooks and making minimal changes. I get the following error somewhere in the process of creating the NewDatabase.

Process SpawnPoolWorker-15: Traceback (most recent call last): File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\pool.py", line 131, in worker put((job, i, result)) File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\queues.py", line 371, in put obj = _ForkingPickler.dumps(obj) File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\pool.py", line 133, in worker wrapped = MaybeEncodingError(e, result[1]) File "C:\Users\hulstmkvd\Anaconda3\envs\premise\lib\multiprocessing\pool.py", line 86, in init self.value = repr(value) MemoryError

Somehow ndb.update_all() manages to continue past this error and continue to extract worksheets and apply strategies.

[romainsacchi]

Hi @Stew-McD @mkvdhulst can you tell me which scenarios you tried to run? To reproduce your issue.

[igt-misc]

@Stew-McD The way I got mine to work was have one script in python 3.10 that imported 3.9.1 consequential unit process into bw, as well as generate biosphere3. From there, I had a separate one in python 3.9 that generated my premise scenarios one at a time - make SSP1_Base_2030, save. Make SSP1_Base_2040, save. Etc. I'm not sure if it's a premise thing or a brightway thing that's making it do this, though, all I've found is that passing multiple scenarios at a time doesn't end well.

[Stew-McD]

🦆 💨 oops. I forgot to specify "system_model= 'consequential' " in the arguments to NewDatabase() It works now 😆

[igt-misc]

@Stew-McD Don't feel bad, I actually emailed Romain because I couldn't get anything to work - turns out I couldn't read and was using the wrong python version.

[Stew-McD]

@igt-misc I think most of the errors will come back via multiprocessing only because that is how the jobs were sent off.

Your memory error is probably from trying to run too many at a time, as you said.

Try expanding your virtual memory, my memory was filling up (16G physical +30G swap/virtual) when running more than 10 at a time. Runs fine now with 60G swap, but it's probably easier just to run separate batches.

@romainsacchi it's curious that the cut-off dbs run fine in Python 3.11, but not the consequential ones. Maybe I'll understand one day when I get to know the code better. 😅

[Stew-McD]

@romainsacchi The code I'm running is here the two "FutureScenarios" scripts

[romainsacchi]

Hi @Stew-McD should I still be looking into ti, or does it work now?

[romainsacchi]

Hi @igt-misc your issue did not re-appear, correct?

[romainsacchi]

Hi @mkvdhulst, how many scenarios did you try to run at once? Does the issue appear when running maybe a smaller set of scenarios, like 5?

[mkvdhulst]

Hi @Stew-McD @mkvdhulst can you tell me which scenarios you tried to run? To reproduce your issue.

@romainsacchi I ran a list of 48 scenarios and applied it to ecoinvent 3.9.1 cut-off. It's a set of 16 specific years between 2023 and 2090 for each of the three SSP2 scenarios of IMAGE. The years are 2023, 2027, 2030, 2033, 2035, 2040, 2045, 2050, 2053, 2060, 2070, 2073, 2075, 2080, 2085, and 2090. I would have loved to try the new ScenarioLink plug-in for this, but the scenarios available are at 5 year intervals, so I would miss some years. Therefore I'm running premise myself.

Last version I was able to run for this set of scenarios was premise 1.5.0-beta3. I wrote all scenarios to the NewDatabase ndb which I then pass to ndb.write_superstructure_db_to_brightway() to obtain a single superstructure database. Its far from ideal, since this takes several hours and any issue along the way wastes hours of computation time, but I managed to pull it off. Somehow premise 1.5.0-beta3 does not drain the RAM, whereas with premise 1.7.0, it immediately maxes out on RAM and CPU. That is without expanding the memory with virtual memory like @Stew-McD did , because I don't know how to do that (just a rookie here). Pointers would be appreciated.

A bit off topic, but it would be useful to have some more information on how to create superstructure databases from databases made with premise. Part of why my calculations take up so much memory is the long list of scenarios that I try to write to a NewDatabase and subsequently to a superstructure database. I'm pretty sure that what I do now is dumb. When I want to add a scenario to my superstructure database, I basically run premise again for the whole list of old + new scenarios, thus redoing a lot of work. One thing that might help is to write each scenario to individual databases. If later I have new scenarios, I would only need to write those to new individual databases. However, I don't yet know how to load those individual database to combine them into a superstructure database. This approach would also require a lot more storage for all these individual databases. Ideally, I would want to add scenarios from NewDatabase to an existing superstructure database. Would that be possible? It would be valuable for coding rookies like me to see some examples in the examples.ipynb of ideal workflows for when you create a lot of scenarios or when you later want to extend your list of scenarios without having to redo all the work.

[romainsacchi]

Hi @mkvdhulst,

OK, I will implement the possibility of not using multiprocessing in v.1.7.1 (I'll let you know when it's out). I will also implement the possibility of generating databases for years in between data points from ScenarioLink. Regarding your last comment, you can use unfold. See some examples here: https://github.com/polca/unfold/blob/main/examples/unfold%20examples.ipynb

Basically, using unfold, you can "fold" premise databases from your brightway project into a datapackage (you will need to also point to the original ecoinvent database in your project). Then, still using unfold, you can "unfold" the datapackage to create a superstructure database. Let me know if that works.

[Stew-McD]

Hi @Stew-McD should I still be looking into it, or does it work now?

It's fine now, I think. Thanks.

Only sometimes I get the same vaccuming database problem that you are already aware of. Trying now with the backported legacy version via pip install git+https://github.com/brightway-lca/brightway2-data.git@bw2legacy

[romainsacchi]

Hi @mkvdhulst, 1.7.1 now allows to disable multiprocessing.

ndb = NewDatabase(
        scenarios=scenarios,
        source_db="ecoinvent 3.9.1 cutoff",
        source_version="3.9.1",
        key=xxxxxxx',
        use_multiprocessing=False
)

Although that means more processing time.

I will now close this issue. If you have a specific issue deriving from this, you can create a new one, or if you think your issue is not solved, you can re-open it.