This merge requests adds a pipeline feature to Nexus so that multiple IDataSources can be chained. This solves two problems:
It is possible to alter catalogs and resources, i.e. with the parallel developed data source Nexus.Sources.Transform users of Nexus can now rename resources, derive units from channel names, assign default groups, etc.
Catalogs can be augmented with additional resources, i.e. two or more data sources can now be responsible for a single catalog. This is useful for plugins that derive data from raw data and where the derived data (i.e. new resources) should be located in the same catalog. This is also useful for cases where data is located in the same folder structure but different file formats. Normally data from all file types in the same folder belong together but will be handled by different plugins / data sources.
To distinguish which data source should handle which data requests, every resource gets an integer property assigned under the path nexus/pipline-position:
This position is set by Nexus when the individual data sources return their resource catalogs. It is then later used to distribute ReadRequests to the corresponding data sources.
At the same time this additional piece of metadata is useful in making the data processing pipeline more tracable so that users can always find out which version of a software and which configuration led to the specific set of data. In future we should add Git support and create a commit every time the configuration changes. The current commit ID will then become part of the catalog metadata (#119).
A frequent change to Nexus source code is the renaming of DataSourceRegistration to Pipeline. This was necessary because we now do not only have a single DataSourceRegistration to provide a set of catalogs but multiple DataSourceRegistrations which compose a pipeline.
There are also many changes regarding to the "format on save" feature, i.e. often useless spaces have been removed. Or I have reformatted some individual LOC without changing their meaning.
Here are some comments to the individually changed files:
.github/workflows:
Use specific pyright version because the new one causes type checking errors (however, this means we need to solve the python type errors in near future, #124)
.vscode/settings.json
Exclude .razor files from editor.formatOnSave because this produces incorrect files
notes/plugin-pipeline.excalidraw
A drawing which shows the pipeline feature, can be ignored
Since we now have a list of data sources (the pipeline), the data source info page has been adapted to display data per data source
src/Nexus.UI/Core/AppState.cs
The CatalogInfo type (contains display info for the UI) had to be adapted so that info about all pipeline members (data sources) can be provided to the UI
mainly renaming from DataSourceRegistration to Pipeline
line 293 (old) / 301 (new): I made the extension method JsonElement.GetStringValue a bit more efficient by reducing the number of string.Split operations which means that now the first parameter is an array instead of a path-like string. This change will occur in other files as well
src/Nexus/API/SourcesController.cs
Previously the user-specific DataSourceRegistration configuration was part of the project.json file in the Nexus configuration folder. This has been factored out and is now part of the user specific folders (also in the Nexus configuration folder):
The file pipelines.json contains all user-configured pipelines and the pipelines itself are managed by the newly created service PipelineService and the file system interaction is handled by the already existing DatabaseService which are both injected into this file (src/Nexus/API/SourcesController.cs).
The REST API code in this file has been adapted to let users interact with pipelines instead of data source registrations.
src/Nexus/API/UsersController.cs
The type InternalDataSourceRegistration became superfluous and has been removed. Now DataSourceRegistration is used everywhere instead
src/Nexus/Core/CatalogContainer.cs
This file mainly follows the name changes and the fact that we now have to handle arrays instead of single object DataSourceRegistrations
src/Nexus/Core/Models_NonPublic.cs
As described above, previously the DataSourceRegistrations were part of project.json which the type UserConfiguration belonged to. Now that DataSourceRegistrations are living in their own pipeline.json files, the type UserConfiguration is not required anymore
src/Nexus/Core/Models_Public.cs
see - same as before comments above
There is now a DataSourcePipeline type which is similar to the old DataSourceRegistration type except that now we have a list of DataSourceRegistrations
This is the core of the changes: Here Nexus has to handle the new pipeline approach, i.e. answers to questions like What to do with multiple GetTimeRange() return values? (because we now have multiple data sources), and more.
The solution for multiple GetAvailability() responses is to calculate the average
The old extension method GetCatalogAsync has been renamed to EnrichCatalogAsync because now every data source gets the catalog returned by the data source which is located earlier in the pipeline. The first data source gets an empty catalog.
Data sources get only read requests passed for resources which belong to the current pipeline position
src/Nexus/Extensions/Sources/Sample.cs
mainly just adapt to other code changes
Line 150 (old) / 149 (new): there now a new tuple parameter called originalResourceName in method ReadAsync. This one became necessary because with the pipeline approach resource IDs can be modified by data source which come later in the pipeline. So the data source which originally provided a resource with a specific ID (= name) cannot rely anymore on the resource ID in the ReadAsync method. Therefore Nexus ensures that every resource has an orignal-name property:
This property can be deliberately set by a data source or - in case the data source doesn't do this - Nexus will do it for you so that this value is never null. So the originalResourceName will now always be part of a ReadRequest.
src/Nexus/Extensions/Writers/Csv.cs
follow previous code changes
src/Nexus/Program.cs
register the PipelineService for DI
src/Nexus/Services/AppStateManager.cs
Data source registrations are now managed by PipelineService, so remove the unnecessary code from here
src/Nexus/Services/CatalogManager.cs
follow previous code changes
src/Nexus/Services/DataControllerService.cs
follow previous code changes
src/Nexus/Services/DataService.cs
follow previous code changes
src/Nexus/Services/DatabaseService.cs
Extend this service with functionality to handle pipeline data
src/Nexus/Services/PipelineService.cs
The pipeline service (handles creation, deletion and retrieval of pipelines per user)
Since Nexus now actively relies on the presence of the original-name resource property, there is a helper method to create it. This already existed in the project Nexus.Sources.StructuredFile but has been moved over into this project
All code which ensures the presence of mandatory catalog and resource properties has been moved over here toa central place
The catalog properties now look a bit different. This is to make the .json object a bit more compact
As mentioned before, the number of string.Split() operations has been reduced to make property access more efficient. Internally catalog and resource properties are represented by a JsonElement and unfortunately it is a bit of work to access nested JSON data. That is the reason why this class exists.
this unit test fixture prepares test data, i.e. it prepares data source registrations (now two instead of one because we want to test the new pipeline behavior)
A data source to be used in the tests and which modifies existing resources and adds a new resource to the catalog. This data source is placed in pipeline position 1, i.e. after the actual data source
This merge requests adds a pipeline feature to Nexus so that multiple
IDataSources can be chained. This solves two problems:To distinguish which data source should handle which data requests, every resource gets an integer property assigned under the path
nexus/pipline-position:This position is set by Nexus when the individual data sources return their resource catalogs. It is then later used to distribute
ReadRequests to the corresponding data sources.At the same time this additional piece of metadata is useful in making the data processing pipeline more tracable so that users can always find out which version of a software and which configuration led to the specific set of data. In future we should add Git support and create a commit every time the configuration changes. The current commit ID will then become part of the catalog metadata (#119).
A frequent change to Nexus source code is the renaming of
DataSourceRegistrationtoPipeline. This was necessary because we now do not only have a singleDataSourceRegistrationto provide a set of catalogs but multipleDataSourceRegistrations which compose a pipeline.There are also many changes regarding to the "format on save" feature, i.e. often useless spaces have been removed. Or I have reformatted some individual LOC without changing their meaning.
Here are some comments to the individually changed files:
.github/workflows:.vscode/settings.json.razorfiles fromeditor.formatOnSavebecause this produces incorrect filesnotes/plugin-pipeline.excalidrawopenapi.jsonsrc/Nexus.UI/Components/CatalogAboutView.razorsrc/Nexus.UI/Core/AppState.csCatalogInfotype (contains display info for the UI) had to be adapted so that info about all pipeline members (data sources) can be provided to the UIsrc/Nexus.UI/Core/NexusDemoClient.cssrc/Nexus.UI/ViewModels/FakeResourceCatalogViewModel.cssrc/Nexus/API/CatalogsController.csDataSourceRegistrationtoPipelineJsonElement.GetStringValuea bit more efficient by reducing the number of string.Split operations which means that now the first parameter is an array instead of a path-like string. This change will occur in other files as wellsrc/Nexus/API/SourcesController.csDataSourceRegistrationconfiguration was part of theproject.jsonfile in the Nexus configuration folder. This has been factored out and is now part of the user specific folders (also in the Nexus configuration folder):The file
pipelines.jsoncontains all user-configured pipelines and the pipelines itself are managed by the newly created servicePipelineServiceand the file system interaction is handled by the already existingDatabaseServicewhich are both injected into this file (src/Nexus/API/SourcesController.cs).The REST API code in this file has been adapted to let users interact with pipelines instead of data source registrations.
src/Nexus/API/UsersController.csInternalDataSourceRegistrationbecame superfluous and has been removed. NowDataSourceRegistrationis used everywhere insteadsrc/Nexus/Core/CatalogContainer.csDataSourceRegistrationssrc/Nexus/Core/Models_NonPublic.csDataSourceRegistrations were part ofproject.jsonwhich the typeUserConfigurationbelonged to. Now thatDataSourceRegistrations are living in their ownpipeline.jsonfiles, the typeUserConfigurationis not required anymoresrc/Nexus/Core/Models_Public.cs- same as beforecomments aboveDataSourcePipelinetype which is similar to the oldDataSourceRegistrationtype except that now we have a list ofDataSourceRegistrationssrc/Nexus/Extensibility/DataSource/DataSourceController.csWhat to do with multiple GetTimeRange() return values?(because we now have multiple data sources), and more.GetCatalogAsynchas been renamed toEnrichCatalogAsyncbecause now every data source gets the catalog returned by the data source which is located earlier in the pipeline. The first data source gets an empty catalog.src/Nexus/Extensions/Sources/Sample.csoriginalResourceNamein methodReadAsync. This one became necessary because with the pipeline approach resource IDs can be modified by data source which come later in the pipeline. So the data source which originally provided a resource with a specific ID (= name) cannot rely anymore on the resource ID in theReadAsyncmethod. Therefore Nexus ensures that every resource has anorignal-nameproperty:This property can be deliberately set by a data source or - in case the data source doesn't do this - Nexus will do it for you so that this value is never
null. So theoriginalResourceNamewill now always be part of aReadRequest.src/Nexus/Extensions/Writers/Csv.cssrc/Nexus/Program.csPipelineServicefor DIsrc/Nexus/Services/AppStateManager.csPipelineService, so remove the unnecessary code from heresrc/Nexus/Services/CatalogManager.cssrc/Nexus/Services/DataControllerService.cssrc/Nexus/Services/DataService.cssrc/Nexus/Services/DatabaseService.cssrc/Nexus/Services/PipelineService.cssrc/Nexus/wwwroot/css/app.csssrc/clients/dotnet-client/NexusClient.g.cssrc/clients/python-client/nexus_api/_nexus_api.pysrc/extensibility/dotnet-extensibility/DataModel/DataModelExtensions.csoriginal-nameresource property, there is a helper method to create it. This already existed in the projectNexus.Sources.StructuredFilebut has been moved over into this projectnew:
old
src/extensibility/dotnet-extensibility/DataModel/PropertiesExtensions.csstring.Split()operations has been reduced to make property access more efficient. Internally catalog and resource properties are represented by aJsonElementand unfortunately it is a bit of work to access nested JSON data. That is the reason why this class exists.src/extensibility/dotnet-extensibility/DataModel/ResourceCatalog.cssrc/extensibility/dotnet-extensibility/Extensibility/DataSource/DataSourceTypes.cssrc/extensibility/dotnet-extensibility/Extensibility/DataSource/IDataSource.csGetCatalogAsynchas been renamed toEnrichCatalogAsyncand the parameters changedsrc/extensibility/python-extensibility/nexus_extensibility/_extensibility_data_source.pytests/Nexus.Tests/DataSource/DataSourceControllerFixture.cstests/Nexus.Tests/DataSource/DataSourceControllerTests.cstests/Nexus.Tests/DataSource/SampleDataSourceTests.cstests/Nexus.Tests/DataSource/TestSource.cs1, i.e. after the actual data sourcetests/Nexus.Tests/Other/CatalogContainersExtensionsTests.cstests/Nexus.Tests/Other/PackageControllerTests.cstests/Nexus.Tests/Services/CatalogManagerTests.cstests/Nexus.Tests/Services/DataControllerServiceTests.cstests/Nexus.Tests/Services/DataServiceTests.cstests/Nexus.Tests/Services/PipelineServiceTests.csPipelineServicetests/Nexus.Tests/Services/TokenServiceTests.cstests/TestExtensionProject/TestDataSource.cs