Deleting parameterized activity does not clean up all related parameter objects

[dgdekoning]

See title. I have a feeling this is caused by how I delete the parameter object, look into this.

Related links to be checked when deleting:

• ParameterizedExchange
• Group
• GroupDependency

[ntropy-esa]

Hello @dgdekoning - I think that the bug you describe in this issue got my project stuck :-(

I have been using the AB to create many activity and project parameters. After a while, I started to clean my project deleting few activities and even a database that was no longer relevant. Today, while re-typing some parameter formulas and duplicating some parametrized-activities (I needed to switch from activity to project parameters), I ran into that error message:

I tried to "re-create" the given parameter in a notebook, using the following code block:

# bw2.parameters.new_project_parameters(wanted, overwrite=True)
data = [
    {'name':'r_ranking_biochar_uses',
    'database':'ranking',
    'code':'d12bdd30428addfaba53fc68e8e072e4',
    'amount':'1',
    'formula':''},
    {'name':'r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4',
    'database':'ranking',
    'code':'d12bdd30428addfaba53fc68e8e072e4',
    'amount':'1',
    'formula':''}]
bw2.parameters.new_activity_parameters(data=data, group='testxxx', overwrite=False) # don't know what group it was (I don't use groups), used both overwrite=True and False
bw2.parameters.recalculate()

I get the same error message in the AB dev (triggered by recalculate() ):

[...]

ActivityParameterDoesNotExist: <Model: ActivityParameter> instance matching query does not exist:
SQL: SELECT "t1"."id", "t1"."group", "t1"."database", "t1"."code", "t1"."name", "t1"."formula", "t1"."amount", "t1"."data" FROM "activityparameter" AS "t1" WHERE ("t1"."group" = ?) LIMIT ? OFFSET ?
Params: ['r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4', 1, 0]

I am pretty sure I don't need that parameter. Is there a way to debug the project by deleting directly in the SQL database the missing entry ? That may be a temporary fix to the bug-

(now I realise I should have maybe created a separate issue)

--- Full error message

IndexError                                Traceback (most recent call last)
~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\peewee.py in get(self, database)
   6806         try:
-> 6807             return clone.execute(database)[0]
   6808         except IndexError:

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\peewee.py in __getitem__(self, item)
   4225             self.fill_cache(item if item > 0 else 0)
-> 4226             return self.row_cache[item]
   4227         else:

IndexError: list index out of range

During handling of the above exception, another exception occurred:

ActivityParameterDoesNotExist             Traceback (most recent call last)
<ipython-input-78-021504db759b> in <module>
     18     'formula':''}]
     19 bw2.parameters.new_activity_parameters(data=data, group='testxxx', overwrite=True)
---> 20 bw2.parameters.recalculate()

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\bw2data\parameters.py in recalculate(self)
   1497             if obj.name in databases or obj.name == 'project':
   1498                 continue
-> 1499             ActivityParameter.recalculate(obj.name)
   1500             ActivityParameter.recalculate_exchanges(obj.name)
   1501 

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\bw2data\parameters.py in recalculate(group)
    789         data = ActivityParameter.load(group)
    790         static = {
--> 791             k: v for k, v in ActivityParameter._static_dependencies(group).items()
    792             if k not in data
    793         }

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\bw2data\parameters.py in _static_dependencies(group)
    616 
    617         Be careful! This could have variables which overlap with local variable names. Designed for internal use."""
--> 618         database = ActivityParameter.get(group=group).database
    619 
    620         chain = [

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\peewee.py in get(cls, *query, **filters)
   6382         if filters:
   6383             sq = sq.filter(**filters)
-> 6384         return sq.get()
   6385 
   6386     @classmethod

~\AppData\Local\conda\conda\envs\bw2\lib\site-packages\peewee.py in get(self, database)
   6810             raise self.model.DoesNotExist('%s instance matching query does '
   6811                                           'not exist:\nSQL: %s\nParams: %s' %
-> 6812                                           (clone.model, sql, params))
   6813 
   6814     @Node.copy

ActivityParameterDoesNotExist: <Model: ActivityParameter> instance matching query does not exist:
SQL: SELECT "t1"."id", "t1"."group", "t1"."database", "t1"."code", "t1"."name", "t1"."formula", "t1"."amount", "t1"."data" FROM "activityparameter" AS "t1" WHERE ("t1"."group" = ?) LIMIT ? OFFSET ?
Params: ['r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4', 1, 0]

[ntropy-esa]

Browsing the closed issues, found #441

In my case: updated AB to latest version, as recommended in #441. Upon start of AB, I get the following messages in the console, letting me know more about the deleted activity and that its parameters will be ignored. Still, parameter system still bugged (impossible to change any parameter value).

I went back to the notebook, to create an activity with that tuple:

fg_db = 'ranking'
db = bw2.Database(fg_db) # loads the data existing, the new scenario will be pushed to it
data = db.load()

code = 'd12bdd30428addfaba53fc68e8e072e4'
new_activity = {'name':'Fixing Activity',
                'code':code,
                'location': 'GLO',
                'unit': 'megajoule',
                'type': 'process',
                'reference product': 'use_2_green-roof',
                'exchanges':[]
               }
data.update([ ((fg_db, code), new_activity) ])

## write that new piece to the database
db.write(data)

Re-loading AB, the message about the deleted activity does not show up in the console. But still, any attempt to change a parameter value raises the same SQL error.

[ntropy-esa]

Third & last message, with a way of fixing a bugged project due to deleted parametrized activities - relevant for #441

Error message to be understood:

ActivityParameterDoesNotExist: <Model: ActivityParameter> instance matching query does not exist:
SQL: SELECT "t1"."id", "t1"."group", "t1"."database", "t1"."code", "t1"."name", "t1"."formula", "t1"."amount", "t1"."data" FROM "activityparameter" AS "t1" WHERE ("t1"."group" = ?) LIMIT ? OFFSET ?
Params: ['r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4', 1, 0]

• The error is triggered when we recalculate all parameters, and it searches for non-properly deleted parameters
• 'Params' does not refer to brightway parameters, but to the parameters of SQL/peewee request (the ? in the request)
• So: group = r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4

Solution: re-create random parameters with the missing group

import brightway2 as bw2

pro = 'cs_Uppsala' # project name
bw2.projects.set_current(pro) # set current project

data = [
    {'name':'FixParam1', # any name
    'database':'ranking', # an existing database
    'code':'d12bdd30428addfaba53fc68e8e072e4', # a different hash for every bugged param, it works by taking the hash given in the error message
    'amount':'1', #any value
    'formula':''} #any value
    ]
bw2.parameters.new_activity_parameters(data=data, group='r_ranking_biochar_uses_d12bdd30428addfaba53fc68e8e072e4', overwrite=True)

bw2.parameters.recalculate() 

If there are several bugged parameters, their name will appear in a new error message (I had 3). Add another block of

data =...
bw2.parameters.new_activity_parameter ... 

before the bw2.parameters.recalculate()

Then, it successfully manages to recalculate parameter values, and the parameter functionalities work again in the GUI.

Hope it helps fixing the management of parameters.

[dgdekoning]

First of all @ntropy-esa, thank you very much for digging so deep into the issue and following the entire path to fixing it.

Due to the disconnectedness of the parameters, it can be very difficult and frustrating to track down all the ways that deleting activities (and even entire databases!) can cause the existing parameters to break down. Part of this I feel is caused by my own implementation of editing and recalculating the parameters within the AB. Basically, I didn't account for the fact that people can delete activities, at all.

My current plan for patching the issue in a way that should make it stay fixed is to hook into the code that notifies you of missing activities and trigger a deletion of the still existing ActivityParameters, ParameterizedExchanges (and the Group, if it is no longer in use).

[dgdekoning]

This should be resoved from version 2.6.2 and onwards.