Project lock makes impossible to parallelize calculations

[aleksandra-kim]

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

After the introduction of the read-only mode for the projects (before was ok) it is not possible anymore to run calculation in parallel even if the projects.enable_writes() is called. attached an example of the file used

[aleksandra-kim]

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

---

Thanks Giuseppe-

This don't have anything to do with temporalis, which doesn't know anything about the write lock. Instead, I think you have one or more databases where the changes haven't been synced to the processed array. What the MC functions in bw2calc do is to call bw2calc.utils.clean_databases before starting any multiprocessing work, as normal LCA calculations don't require writes, and so work just find. See e.g. https://bitbucket.org/cmutel/brightway2-calc/src/45d7f0f473d2bd732978144f0ccbe69cd4219f7c/bw2calc/monte_carlo.py?at=default&fileviewer=file-view-default#monte_carlo.py-132.

Could you try this and see if it resolves the issue? Otherwise, can you provide more of a traceback on the exact error raised?

[aleksandra-kim]

Original comment by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

Hey Chris,

I also found strange that the calculations were writing something to the databases, anyway, I tried but seems that what you mentioned is not the problem.

This is the traceback before adding bw2calc.utils.clean_databases() (that actually seems to already call clean_databases())

#!python
python multiproc_dyn_all_imp_cat_5.py
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3

did you check current.project?

3 FUs per 2 impact categories to analyse

Chunks left in static: 3
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3
/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2data/project.py:125: UserWarning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

  warnings.warn(READ_ONLY_PROJECT)
/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2data/project.py:125: UserWarning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

  warnings.warn(READ_ONLY_PROJECT)
Warning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

Warning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

Warning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "multiproc_dyn_all_imp_cat_5.py", line 61, in stat_lca
    lca_t = bw2calc.LCA(demand=FU, method=methods)
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2calc/lca.py", line 52, in __init__
    clean_databases()
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2calc/utils.py", line 64, in clean_databases
    databases.clean()
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2data/project.py", line 299, in writable_project
    raise ReadOnlyProject(READ_ONLY_PROJECT)
bw2data.errors.ReadOnlyProject: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "multiproc_dyn_all_imp_cat_5.py", line 118, in <module>
    stat_multi=[x for x in pool_stat_multi.get()] 
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
bw2data.errors.ReadOnlyProject: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

and after

#!python

python multiproc_dyn_all_imp_cat_5.py
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3

did you check current.project?

2 FUs per 2 impact categories to analyse

Chunks left in static: 2
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3
Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/bw2_joana/Brightway3
/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2data/project.py:125: UserWarning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

  warnings.warn(READ_ONLY_PROJECT)
Warning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

Warning: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "multiproc_dyn_all_imp_cat_5.py", line 58, in stat_lca
    bw2calc.utils.clean_databases() ###
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2calc/utils.py", line 64, in clean_databases
    databases.clean()
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/site-packages/bw2data/project.py", line 299, in writable_project
    raise ReadOnlyProject(READ_ONLY_PROJECT)
bw2data.errors.ReadOnlyProject: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "multiproc_dyn_all_imp_cat_5.py", line 125, in <module>
    stat_multi=[x for x in pool_stat_multi.get()] 
  File "/home/giuseppec/miniconda3/envs/bw2_dev/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
bw2data.errors.ReadOnlyProject: 
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

Here you will find both the file I used to run the calculations (i.e. the previous one but with bw2calc.utils.clean_databases() added) plus the bw2 folder with the databases if you need to test on your pc:

P.S: just for completeness, I set the BRIGHTWAY2_DIR in dropbox so I can do everything in my laptop and then launch the calculations (that should take a couple of days) in another laptop that use this data directory....I was just wondering if during the sincronizations dropbox did not mess up something (maybe makes not sense what I said, in any case I log and sync dropbox only at the end when I finished all the task of writing of the dbs)

[aleksandra-kim]

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

---

Thanks Giuseppe, I will look into this tomorrow. I think that the problem is that databases.clean raises a ReadOnlyProject even if it doesn't need to actually write anything, which is obviously incorrect...

[aleksandra-kim]

Original comment by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

Thanks Chris, would be great if you could give this a look tomorrow that have to send tun quite urgently this runs....after long time of work I though could eventually run things and when I got this error yesterday was almost crying :)

[aleksandra-kim]

Original comment by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

Hey Chris, I changed the link to the dropbox folder, was working on that folder and added things that could confuse you, now the link above is correct (the folder bw2_dev contains everything) Ciao

[aleksandra-kim]

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

---

.clean() shouldn't raise errors if no writes are needed. Fixes #32

[aleksandra-kim]

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

---

Fixed in new 2.0.dev10 release.

[aleksandra-kim]

Original comment by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

Hey Chris,

I updated 2.0.dev10 to from pip and tried again but, after the database has been updated i.e.

#!python
$ python multiproc_dyn_all_imp_cat_5.py

Using environment variable BRIGHTWAY2_DIR for data directory:
/home/giuseppec/Dropbox/multiprocess/bw2_dev
Applying automatic update: 2.0-2 database search directories
Reindexing database FORMIT_forest
Reindexing database FORMIT_forest_FU

I got exactly the same error as above :S

hope did not make any mistake

P.S: again everything is in the dropbox folder above if needed

[aleksandra-kim]

Original comment by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

---

now works!