AB not pointing to custom BRIGHTWAY2_DIR

[cardosan]

In BW2 it is possible to set a custom BRIGHTWAY2_DIR and I have done it to have all the data on a server. When launching bw2 from the terminal it points to the correct path but, when I open AB, it points to the default BRIGHTWAY2_DIR location i.e. C:\Users\USER\AppData\Local\pylca\Brightway3. I know it can be changed on the settings but would expect (and I think makes more sense ) that this happens by default when the custom data directory path is changed in an environment.

To test you can use the following batch file (changed extention to .txt just to be able to attach) that creates a new conda environment, install ab and set the custom BRIGHTWAY2_DIR, just change the fields in line 3 and 24

AB_install.txt

[StpdFox]

Oh thanks for including the batch file! I'll add it to the readme for people who want a custom installation! Custom locations is one the the things that'll come when we add an actual installation wizard instead of using conda, but this is a great fix in the meantime

[cardosan]

HI @StpdFox, the prob here is that the batch file does not work as it should since, when opening AB, AB override the settings in the batch file, so no really a temporary fix :S

[StpdFox]

Ah , Sorry i think i misread it then. But i checked for the AB option to change te BW2 Directory and it's in the settings menu. Have you tried this? Or am i misunderstanding the issue?

[cardosan]

No worries, maybe I was not clear enough ;)

what I wanted is not to have to set the BRIGHTWAY2_DIR manually in the setting of AB every time it is used (i.e. what you showed) but rather have the BRIGHTWAY2_DIR set persistently within the environment where AB (and obviously BW2) is installed. This work when you open bw2 via terminal (obviously) but not when AB is opened.

From some reasons it seems that project settings are reset when I open AB. As you can see below from the outputs I get when opening AB, first it points to the right directory but then switch to the default (wrong) on.

(ab_y) C:\WINDOWS\system32>activity-browser

######CORRECT#####
Using environment variable BRIGHTWAY2_DIR for data directory:
\\vito.local\vito\Unit_SEB\1_MACS\Models and databases\BW2
-hiding tab: History

DevTools listening on ws://127.0.0.1:3999/devtools/browser/16d6fb93-8e18-4508-9f3c-60cfb0b813fd
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results

######STILL CORRECT#####
Reset project settings directory to: \\vito.local\vito\Unit_SEB\1_MACS\Models and databases\BW2\default.c21f969b5f03d33d43e04f8f136e7682 #still correct
Reset metadata.
Loading user settings:

######CHANGED TO DEFAULT#####
Using environment variable BRIGHTWAY2_DIR for data directory:
\\vito.local\vito\Unit_SEB\1_MACS\Models and databases\BW2
Loaded brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Reset project settings directory to: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Reset metadata.
Loaded project: default
Brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Brightway2 active project: default
Qt Version: 5.12.9

Hope is clearer now

P.S: obviously feel free to use the batch file for whatever purpose it might be useful

[cardosan]

The problem here is the ABsetting file , where AB looks for settting the bw2 directory at opening..

more specifically : 1) AB always points to the AB setting file at opening, regardless of the environment you are in (so not environment but user specific) 2) AB look at it for setting the initial bw2 directory (i.e. custom_bw_dir in the file) when you lunch it, disregarding any BW2 setting (i.e. BRIGHTWAY2_DIR) if present 3) if the ABsetting file is not present, it is created setting as custom_bw_dir the current BRIGHTWAY2_DIR.

This situation is, in my opinion, suboptimal cause

a) the setting should obviously be environment specific. One might want to have different AB/BW2 installations for different projects/needs. E.g. I would have a installation point to the group bw2 database in the server and the another one local

b) the BW2 setting are disregarded, while AB and BW2 should go along in terms of setting since one of the typical use cases of AB would be to do switch between it and BW2 depending on the needs.

What I was actually experiencing is that the custom BRIGHTWAY2_DIR was set after I installed for the first time AB and when setting file was created (see point 3) and thus at opening AB keept pointing to the default BRIGHTWAY2_DIR

It seems that this file is only used for setting the bw2 dir and the startup project. Instead of creating the custom_bw_dir in ABsetting, I would propose to directly change the BRIGHTWAY2_DIR when the user changes the brightway dir in the setting file of AB so to have have consistency between BW2 and AB behaviour and keep in the file only the settings related to the project .

[cardosan]

as agreed upon, approach to deal with this will be:

1. get rid of the the custom_bw_dir in the AB_settings.json file and just work with BRIGHTWAY2_DIR,
2. default location to be used is that of the environmental variable BRIGHTWAY2_DIR (set from appdirs during installation )
3. to use another folder to store data in anenvironment the BRIGHTWAY2_DIR has to be changed in brightway directly following this approach rather than AB being a rather geeky necessity
4. AB points to existing BRIGHTWAY2_DIR (either the global or the one in the environmental variable set as before)
5. AB shows dropdownlist (in settings->Brightway dir?) with the available i.e. previously set custom BRIGHTWAY2_DIR plus the default ones

See also #696

P.S: unsure I got correctly point 5

[Zoophobus]

@cardosan Hi, so we have been going over this a bit and encountered (of course) a couple of limiting issues when it came to reading the BRIGHTWAY2_DIR environment variable. E.g. catching those instances where it is being overwritten/set in any particular system. It required the use of the conda executable, which made debugging within the IDE highly problematic/impossible. We have gone for a slightly different approach.

Unfortunately, for yourself I'm assuming, we have retained the AB_settings.json file, however we've amended the structure. Now this file contains two strings (a default for the environment and another for the project) and a list of strings for the different environments. In this the user can select multiple different environments in their system using the browse button (as before), each new environment then being saved to the AB_settings.json file. On starting the AB these settings are then read and a drop down list is created with the specified values in the settings file. This means that the user will still have some work, but switching between the different environments should be much quicker

[cardosan]

Hi @Zoophobus I tried to test it but with limited success.

I am assuming the dev version I tried i.e.:

# Name                    Version                   Build  Channel
activity-browser-dev      2022.12.22                 py_0    bsteubing

contains the implemented changes since I could see that the strcture of AB_settings.json changed like that

{
    "current_bw_dir": "C:\\Windows\\System32",
    "custom_bw_dirs": [
        "C:\\Users\\CARDELLG\\AppData\\Local\\pylca\\Brightway3",
        "C:\\Windows\\System32"
    ],
    "startup_project": "default"
}

In thi case no matter what folder I choose, I get the following error:

DevTools listening on ws://127.0.0.1:3999/devtools/browser/c8c545c6-6f2a-4e91-b90c-2fa1fdd25155
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results
Reset project settings directory to: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Loading user settings:
dirpath already loaded
Brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Brightway2 active project: default
Qt Version: 5.15.4
Remote debugging server started successfully. Try pointing a Chromium-based browser to http://127.0.0.1:3999
QWizardPage::addField: Duplicate field 'current_bw_dir'
Traceback (most recent call last):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\controllers\project.py", line 31, in switch_brightway2_dir_path
    if bc.switch_brightway2_dir(dirpath):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\commontasks.py", line 94, in switch_brightway2_dir
    os.mkdir(bw.projects._base_logs_dir)
PermissionError: [WinError 5] Access is denied: 'C:\\Windows\\System32\\logs'

[cardosan]

Small update: If I run AB (i.e. the prompt were I launch it) as admin I get this error :

(ab_dev) C:\WINDOWS\system32>activity-browser
-hiding tab: History

DevTools listening on ws://127.0.0.1:3999/devtools/browser/0f0331e4-bec4-4023-9dc1-233e719b0049
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results
Reset project settings directory to: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Loading user settings:
dirpath already loaded
Brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Brightway2 active project: default
Qt Version: 5.15.4
Remote debugging server started successfully. Try pointing a Chromium-based browser to http://127.0.0.1:3999
QWizardPage::addField: Duplicate field 'current_bw_dir'
Loaded brightway2 data directory: C:\Users\CARDELLG\Downloads\xl2bb
Traceback (most recent call last):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\controllers\project.py", line 32, in switch_brightway2_dir_path
    self.change_project(ab_settings.startup_project, reload=True)
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\controllers\project.py", line 49, in change_project
    assert name, "No project name given."
AssertionError: No project name given.

guess AB is assuming the folder contains a projects.db file, which is typically not the case when point to a new (empty) folder.

[Zoophobus]

Hi @Zoophobus I tried to test it but with limited success.

I am assuming the dev version I tried i.e.:

# Name                    Version                   Build  Channel
activity-browser-dev      2022.12.22                 py_0    bsteubing

contains the implemented changes since I could see that the strcture of AB_settings.json changed like that

{
    "current_bw_dir": "C:\\Windows\\System32",
    "custom_bw_dirs": [
        "C:\\Users\\CARDELLG\\AppData\\Local\\pylca\\Brightway3",
        "C:\\Windows\\System32"
    ],
    "startup_project": "default"
}

In thi case no matter what folder I choose, I get the following error:

DevTools listening on ws://127.0.0.1:3999/devtools/browser/c8c545c6-6f2a-4e91-b90c-2fa1fdd25155
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results
Reset project settings directory to: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Loading user settings:
dirpath already loaded
Brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Brightway2 active project: default
Qt Version: 5.15.4
Remote debugging server started successfully. Try pointing a Chromium-based browser to http://127.0.0.1:3999
QWizardPage::addField: Duplicate field 'current_bw_dir'
Traceback (most recent call last):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\controllers\project.py", line 31, in switch_brightway2_dir_path
    if bc.switch_brightway2_dir(dirpath):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\commontasks.py", line 94, in switch_brightway2_dir
    os.mkdir(bw.projects._base_logs_dir)
PermissionError: [WinError 5] Access is denied: 'C:\\Windows\\System32\\logs'

Hi Cardosan, do you mind my asking why you are trying to use the Windows\System32 folder? For my understanding (now really quite limited understanding) this would be a protected system folder, one that the typical user will not have write access for, which should also be the case for the AB.

[Zoophobus]

guess AB is assuming the folder contains a projects.db file, which is typically not the case when point to a new (empty) folder.

Hmmm, okay, I'm going to have to go over this again, obviously I've missed something here, sorry for that

[cardosan]

Hi @Zoophobus I tried to test it but with limited success. I am assuming the dev version I tried i.e.:

# Name                    Version                   Build  Channel
activity-browser-dev      2022.12.22                 py_0    bsteubing

contains the implemented changes since I could see that the strcture of AB_settings.json changed like that

{
    "current_bw_dir": "C:\\Windows\\System32",
    "custom_bw_dirs": [
        "C:\\Users\\CARDELLG\\AppData\\Local\\pylca\\Brightway3",
        "C:\\Windows\\System32"
    ],
    "startup_project": "default"
}

In thi case no matter what folder I choose, I get the following error:

DevTools listening on ws://127.0.0.1:3999/devtools/browser/c8c545c6-6f2a-4e91-b90c-2fa1fdd25155
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results
Reset project settings directory to: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Loading user settings:
dirpath already loaded
Brightway2 data directory: C:\Users\CARDELLG\AppData\Local\pylca\Brightway3
Brightway2 active project: default
Qt Version: 5.15.4
Remote debugging server started successfully. Try pointing a Chromium-based browser to http://127.0.0.1:3999
QWizardPage::addField: Duplicate field 'current_bw_dir'
Traceback (most recent call last):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\controllers\project.py", line 31, in switch_brightway2_dir_path
    if bc.switch_brightway2_dir(dirpath):
  File "C:\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\commontasks.py", line 94, in switch_brightway2_dir
    os.mkdir(bw.projects._base_logs_dir)
PermissionError: [WinError 5] Access is denied: 'C:\\Windows\\System32\\logs'

Hi Cardosan, do you mind my asking why you are trying to use the Windows\System32 folder? For my understanding (now really quite limited understanding) this would be a protected system folder, one that the typical user will not have write access for, which should also be the case for the AB.

I was also puzzled by the access denied to that folder since I was not pointing to that path ....I have thus tried to set the custom folder in different locations (my institution server, Documents, C:\Users\myusername\etc.) always with the same exact error.

[cardosan]

@cardosan Hi, so we have been going over this a bit and encountered (of course) a couple of limiting issues when it came to reading the BRIGHTWAY2_DIR environment variable. E.g. catching those instances where it is being overwritten/set in any particular system. It required the use of the conda executable, which made debugging within the IDE highly problematic/impossible. We have gone for a slightly different approach.

Frankly I have not fully understood where the problem is here. I thought that the approach used in check_bw2dir function would be ok to do this. Anwyay, happy also to have a chat to clarify unclear points (if any) and help if I can (I spent quite some time on trying to deal with this issues at the time )

[Zoophobus]

Hi @Zoophobus I tried to test it but with limited success.

I am assuming the dev version I tried i.e.: It should be alright to use the dev version from B.Steubings conda account, however we're going to start to recommend people to use the conda-forge version, it should be more reliable with maintaining the different package versions. Having said that the updates I am currently pushing do first go to the dev version.

Also sorry for the delay, I was going through the code for this issue today and got a little stuck running through the code around the construction of the databases in AB, I didn't realise that bw is "smart" and loads projects automatically (I was always assuming it was "dumb" and doing things manually). Having written that I have pushed some updates to a local branch (bug_585) on my local AB fork, I'll merge it with the main branch once the tests are done.

I was also puzzled by the access denied to that folder since I was not pointing to that path ....I have thus tried to set the custom folder in different locations (my institution server, Documents, C:\Users\myusername\etc.) always with the same exact error.

I think we should arrange a call, I've just checked the latest version of AB on windows (it was a bit of a struggle for me to be honest, I'm normally avoiding Windows) and I didn't get this error. I'm not sure how much I can really help but maybe together we can get closer to a solution

[cardosan]

I reinstalled conda and AB and could solve the protected system folder issue and test it. some feedbacks here:

1. I find the drop-down list really useful (+1!),
2. If I do not remember wrong you are already aware, but it is not possible to set in AB a Brightway Diran empty folder (it assumes it has been already initialized by BW2, which is not necessarily the case)
3. I still think the inconsistent behaviour between AB and BW2 and the fact that the settings are non environmnet-specific are still a limitation. E.g. now that that different installations are needed for different ei versions might make people wanting to 1 enviro (and anunderlying folder with databases) for older ei version, and another with the more recent ei linked to another database folder and keeop both isolated. It is stil possible to switch manually between enviro/db folders but this increase also the chances of making mistakes (e.g. creating the initial default settings with the wrong bw2io version)

[Zoophobus]

Hi, so thanks for the first point and the second! So this might be something else that I'll need to check, on Linux I don't get this behaviour, I've attached an image of the log messages (you can find them in the AB debug window) that shows the execution of a series of bw methods that setup the environment.

Could you provide a similar view of what happens for you?

As for the third point, we have had quite a few discussions on this. I have argued for creating a warning message for these environments, however it was pointed out that this is really something for the brightway level of code. Then there's the part where if they can solve the issue between the EI versions, well then this is also moot. Basically I'm not sure at the moment how to manage this.

[marc-vdm]

Closing as outdated