odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
807 stars 261 forks source link

rest call to /open-metadata/admin-services/users/{userId}/configurations never returns anything #5576

Closed davidradl closed 3 years ago

davidradl commented 3 years ago

rest call to /open-metadata/admin-services/users/{userId}/configurations never returns anything.

The reason appears to be that the logic was no updated when we moved the server configs out of the top folder. It is looking in the current top folder for a match on a pattern of "data/servers/{0}/config/{0}.config". This template is looking to identify a file deeper than the current folder. A complexity also is that this filename is overridable, so the fix needs to handle that.

planetf1 commented 3 years ago

I did make a move to where we store configs last year....

The principle was to store all transient data under 'data', relative to the current working directory of the process (platform). This made it easy to add a 'volume' in container environments to store this data.

The 'relative' nature is important -- for example in our containers/assemblies we tend to run from a top level which contains 'server'. 'clients' etc, whilst often in a dev environment we use the top of the source tree. Either should work, but obviously one won't find the other.

Looking at my current egeria where I last ran the coco dev environment, I see files of the form

$ find ./data -name 'cocoMDS1.config'                                     [12:41:27]
./data/servers/cocoMDS1/config/cocoMDS1.config

I posted a recursive listing to https://gist.github.com/6df6f6d652a19cd37d6fc977d9f15229

You mention in the text that we are looking for 'data/servers/{0}/config/{0}.config' If {0} resolved to 'cocoMDS1' in my config the file would be found.

So what don't you have, and what do you think you should have? I may be missing what the problem is here (apart from the whole configuration question ...) Or is it an implementation issue where file was looked for rather than resolving a path?

planetf1 commented 3 years ago

I do concur that the output from the rest call is empty though

davidradl commented 3 years ago

In file https://github.com/odpi/egeria/blob/master/open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/EncryptedFileBasedServerConfigStoreConnector.java

In the code `retrieveAllServerConfigs() { final String methodName = "retrieveAllServerConfigs"; Set omagServerConfigSet = new HashSet<>(); try (Stream list = Files.list(Paths.get("."))) { // we need to use the configStoreTemplateName to pick up any files that match this shape. // this template might have inserts in

        String templateString = getStoreTemplateName();
          Set<String> fileNames = list.map(x -> x.toString())
                .filter(f -> isFileNameAConfig(f, templateString)).collect(Collectors.toSet());`

The Files.list command goes down the files in the current directory; t does not match the nested content, so it never finds anything.

Unfortunately using a * wild card did not work @mandy-chessell

I have a fix I have working that splits up the template string and then looks for the content where the wild cards are. So it can create a valid list of fileNames that match the template It handles one or 2 inserts and rejects 0 or >2 inserts in the template.

I have amended the default constant slights to prepend with ./.

 private static final String INSERT_FOR_FILENAME_TEMPLATE = "{0}";
 private static final String DEFAULT_FILENAME_TEMPLATE    = "./data/servers/" +INSERT_FOR_FILENAME_TEMPLATE + "/config/" +INSERT_FOR_FILENAME_TEMPLATE + ".config";

Does this make sense?

planetf1 commented 3 years ago

@davidradl That looks good to me. I would have broken this api when I did 1a9ba66bd8fa0b5dc45942977e1e30ca869ea5ce :-(