astarte-platform / astarte

Core Astarte Repository
https://docs.astarte-platform.org/
Apache License 2.0
240 stars 46 forks source link

Edit Astarte keyspaces #909

Closed eddbbt closed 5 months ago

eddbbt commented 7 months ago

This pr enable multiple astarte instances to share a common db via keyspaces separation. With the recent deploy on google cloud marketplace this is likely to be a common scenario with shared and low tiers.

At the actual state this project does not allow multiple instances to share the same db and thus is more prone to name conflicts for realms and organizations, due to the shared keyspaces and tables.

DB connector and queries have been edited to differentate beetween keyspaces via an env variable (called an astarte_instance_id), this way each deploy creates it's own astarte keyspace and for each realm, a keyspace wich name's prefixed by its own astarte_instance_id. Proper escaping and validation have been setup to assure there will be no conflicts and no incompatibilities with the selected scylla/cassandra db.

All tests for all astarte modules have been fixed and optimized for the new working logic, it was not necessary to change tests scope or goals. Also the testing db has been corrected.

This pr is heavily dependant on:

There are dependencies pointing at the forked repos above, those needs to be changed before release

Further info can be found in notion

resolves https://github.com/astarte-platform/astarte/issues/924

codecov[bot] commented 7 months ago

Codecov Report

Attention: Patch coverage is 77.29258% with 52 lines in your changes are missing coverage. Please review.

Project coverage is 68.40%. Comparing base (1b98a40) to head (db8f066). Report is 2 commits behind head on master.

:exclamation: Current head db8f066 differs from pull request most recent head 7071312. Consider uploading reports for the commit 7071312 to get more accurate results

Files Patch % Lines
...ib/astarte_data_updater_plant/data_updater/impl.ex 77.96% 13 Missing :warning:
...e_housekeeping/lib/astarte_housekeeping/queries.ex 88.00% 6 Missing :warning:
...management/lib/astarte_realm_management/queries.ex 77.77% 6 Missing :warning:
..._housekeeping/lib/astarte_housekeeping/migrator.ex 0.00% 4 Missing :warning:
...appengine_api/lib/astarte_appengine_api/queries.ex 0.00% 3 Missing :warning:
...ine_api/lib/astarte_appengine_api/device/device.ex 89.47% 2 Missing :warning:
...astarte_data_updater_plant/data_updater/queries.ex 0.00% 2 Missing :warning:
..._housekeeping/lib/astarte_housekeeping/c_system.ex 0.00% 2 Missing :warning:
...tarte_realm_management/device_removal/scheduler.ex 0.00% 2 Missing :warning:
...te_appengine_api/utils/astarte_instance_id_type.ex 50.00% 1 Missing :warning:
... and 11 more
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #909 +/- ## ========================================== + Coverage 62.05% 68.40% +6.34% ========================================== Files 57 282 +225 Lines 1389 7397 +6008 ========================================== + Hits 862 5060 +4198 - Misses 527 2337 +1810 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

osmanhadzic commented 7 months ago

WIP

Please rephrase both the commit message and the title. In general, commits should describe:

Why a change is being made? How does the change address the issue? What effects does the change have? Are there any limitations of the current code? For the PR, you can take a look at https://www.pullrequest.com/blog/writing-a-great-pull-request-description/

eddbbt commented 7 months ago

thank you @osmanhadzic, I appreciate your advice, It will be a priority to write a meaningful description and title after we figure out everything, but please consider this pr is currently a work in progress even at a design level