Update environment cache performance is slow

In a Calm scenario with 100+ projects, updating or creating new Environments takes hours. Calm updates projects and environments after every excecution. That means in a for_loop, the process takes a huge amount of time. Is there a way to skip the cache update? or a better way to loop through multiple projects?

We also often see the error below. It does not break the loop, but slows the process down even more.

[2021-12-02 19:33:14] [INFO] [calm.dsl.cli.environments:309] Updating projects and environments cache ... Traceback (most recent call last): File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2981, in execute_sql cursor.execute(sql, params or ()) sqlite3.IntegrityError: UNIQUE constraint failed: environmentcache.name, environmentcache.uuid

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users**\calm-dsl\venv\Scripts\calm-script.py", line 11, in load_entry_point('calm.dsl', 'console_scripts', 'calm')() File "C:\Users**\calm-dsl\venv\lib\site-packages\click\core.py", line 829, in call return self.main(args, kwargs) File "C:\Users\*\calm-dsl\venv\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "c:\users**\calm-dsl\calm\dsl\cli\utils.py", line 127, in invoke return super().invoke(ctx) File "C:\Users**\calm-dsl\venv\lib\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "c:\users**\calm-dsl\calm\dsl\cli\utils.py", line 116, in invoke return super().invoke(ctx) File "C:\Users**\calm-dsl\venv\lib\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users**\calm-dsl\venv\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "C:\Users**\calm-dsl\venv\lib\site-packages\click\core.py", line 610, in invoke return callback(args, kwargs) File "c:\users\*\calm-dsl\calm\dsl\cli\environment_commands.py", line 99, in _update_environment update_environment_from_dsl_file(env_name, env_file, project_name) File "c:\users**\calm-dsl\calm\dsl\cli\environments.py", line 311, in update_environment_from_dsl_file Cache.sync_table(cache_type=CACHE.ENTITY.ENVIRONMENT) File "c:\users**\calm-dsl\calm\dsl\store\cache.py", line 164, in sync_table cache_table.sync() File "c:\users**\calm-dsl\calm\dsl\db\table_config.py", line 873, in sync cls.create_entry( File "c:\users**\calm-dsl\calm\dsl\db\table_config.py", line 882, in create_entry super().create( File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 6122, in create inst.save(force_insert=True) File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 6311, in save pk = self.insert(field_dict).execute() File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 1796, in inner return method(self, database, args, kwargs) File "C:\Users\*\calm-dsl\venv\lib\site-packages\peewee.py", line 1867, in execute return self._execute(database) File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2599, in _execute return super(Insert, self)._execute(database) File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2336, in _execute cursor = database.execute(self) File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2994, in execute return self.execute_sql(sql, params, commit=commit) File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2988, in execute_sql self.commit() File "C:\Users**\calm-dsl\venv\lib\site-packages\peewee.py", line 2759, in exit reraise(new_type, new_type(exc_args), traceback) File "C:\Users\*\calm-dsl\venv\lib\site-packages\peewee.py", line 183, in reraise raise value.with_traceback(tb) File "C:\Users****\calm-dsl\venv\lib\site-packages\peewee.py", line 2981, in execute_sql cursor.execute(sql, params or ()) peewee.IntegrityError: UNIQUE constraint failed: environmentcache.name, environmentcache.uuid

nutanix / calm-dsl

Update environment cache performance is slow #203