Support for Private Genesis Instance

JoelProminic commented 11 months ago

@dpastov has an initial implementation for private Genesis instances, so that a customer a set of applications for their internal use. I'll fill in some details for this later

JoelProminic commented 11 months ago

This page has the details for private Genesis directories, with additional documentation here. I went through the setup manually, but I'm discussing the issues I found separately with @dpastov. Some quick notes:

This requires Genesis 0.6.19, which is not released to production yet

The command syntax is:

tell Genesis origin %url% %password% install %app%

Parameter	Description
`%url%`	The URL to the Genesis Directory database. This could be a local or remote database. The `domino.%server%` domain doesn't work for local servers, because this is not configured in the hosts file. Even if you update the hosts file, this will not go through the proxy, so it wil trigger an SSL error.
`%password%`	The password for a private directory or `-` for a public directory
`%app%`	the app ID

Previously the install command was returned by the Genesis API. If I keep this logic, I'll need to make sure the command includes these parameters: origin %url% %password%

Note that the customer repositories are not configured in Genesis, so the GenesisRead agent doesn't have a convenient configuration file to read to identify the vaialble servers. So, I will probably want to add a new form and view to SuperHumanPortal.nsf to track the private Genesis directories that should be available in the Super.Human.Portal interface. I would expect the user to assign a label to the configured directory so that it can be referenced with a more user-friendly name. The full URL (and password) could then be looked up and populated for the command.

In order to read the applications from a private Genesis catalog, I should be able to a similar JSON API to what I currently use for the official directory. @dpastov needs to confirm this.

JoelProminic commented 10 months ago

For the UI, we'd like to add a new property to GenesisRead ("directory"?) for a new column to show the non-default directories. This property will be passed as a parameter to GenesisInstall:

To support adding additional Genesis Directories, we'll add a new section called "Additional Directories". If we can't add it as a child of "Genesis App" in the sidebar, we can consider other arrangements, but it should be clear that the new interface is meant to extend Genesis App.

The new interface should be a mostly straightforward CRUD interface for the new GenesisDirectory* agents (not yet created).

Note that the password can only be set or cleared - it will not be returned by GenesisDirectoryRead.

JoelProminic commented 10 months ago

I have the agent updates pushed in the features/issue_35_private_genesis_instance branch.

I still need to:

Do some tests with the API
Writeup some proper documentation
Send a test database copy with the new forms and views.
UPDATE: I forgot about the special logic for handling the password. This may take a bit longer to implement.

@piotrzarzycki21, I'll try to get this finished early today. In the meantime, the generated documentation is available here

JoelProminic commented 10 months ago

I have updated the GenesisDirectory CRUD agents on the features/issue_35_private_genesis_instance branch, and I sent a new copy of the database for testing.

GenesisDirectoryRead

Example URL: https://domino.demo.startcloud.com/SuperHumanPortal.nsf/GenesisDirectoryRead?OpenAgent Example JSON:

{
  "documents": [
    {
      "DominoUniversalID": "0B2D...",
      "private": "true",
      "label": "Test",
      "url": "TestURL"
    },
  ],
  "errorMessage": "",
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated"
}

Property	Description
label	A unique name (currently not enforced) for the directory. This will be added to the GenesisRead and GenesisInstall agents later to indicate the repository to use.
url	The URL for the repository, including the database. Example: `https://domino.demo.startcloud.com/gdp1.nsf/`
private	"true" if repository is considered private (hidden `password` property is not blank), any other values means it is public
DominoUniversalID	Standard unique identifier to use for Update and Delete agents

GenesisDirectoryCreate

Example URL: https://domino.demo.startcloud.com/SuperHumanPortal.nsf/GenesisDirectoryCreate?OpenAgent&label=Test&url=https://domino.demo.startcloud.com/gc1.nsf&password=mypass Example response:

{
  "document": {
    "DominoUniversalID": "0B2D...",
    "private": "true",
    "label": "Test",
    "url": "TestURL"
  },
  "errorMessage": "",
  "validationErrors": [],
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated"
}

Parameter	Description
label	See above
url	See above
password	The password for a private repository, or "" if no password is required.

Note that there is no parameter for private - this is currently computed based on the value of password The computed value will be returned in document

GenesisDirectoryUpdate

Example URL: https://domino.demo.startcloud.com/SuperHumanPortal.nsf/GenesisDirectoryUpdate?OpenAgent&DominoUniversalID=A761...&label=Test3&url=TestURL&password= Example response:

{
  "errorMessage": "",
  "validationErrors": [],
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated"
}

Parameter	Description
DominoUniversalID	Key from Read or Create agent
label	See above
url	See above
password	An updated password. If the user doesn't try to change the password, don't pass this parameter, or it will overwrite the existing value. Pass "" to switch this to be a public repository.

GenesisDirectoryDelete

Example URL: https://domino.demo.startcloud.com/SuperHumanPortal.nsf/GenesisDirectoryUpdate?OpenAgent&DominoUniversalID=A761...&label=Test3&url=TestURL&password= Example response:

{
  "errorMessage": "",
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated"
}

Parameter	Description
DominoUniversalID	Key from Read or Create agent

JoelProminic commented 10 months ago

@piotrzarzycki21, I also updated the GenesisRead agent to add the directory property. This is intended to match the label for an additional GenesisDirectory, or a default value (currently "Default") for the default/central Genesis Directory.

{
  "errorMessage": "",
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated",
  "apps": [
    {
      "access": {
        "description": "Prominic.NET - demo javaaddin",
        "links": []
      },
      "AppID": "javaaddindemo",
      "DetailsURL": "https://genesis.directory/apps/javaaddindemo",
      "InstallTimeS": 15,
      "Installed": true,
      "Label": "JavaAddin - Demo",
      "InstallCommand": "install javaaddindemo",
      "directory": "Default"
    },

I'm thinking we may change the default directory value to "" to make the custom directory entries stand out more in the datagrid. I set a value for now to make testing clearer, but we'll do a test with a blank value later to see how it looks.

The directory parameter for the selected application should be passed to the GenesisInstall agent. The new functionality is not implemented yet, but you can add the new parameter currently without breaking the agent.

piotrzarzycki21 commented 10 months ago

{
  "document": {
    "DominoUniversalID": "0B2D...",
    "private": "true",
    "label": "Test",
    "url": "TestURL"
  },
  "errorMessage": "",
  "validationErrors": [],
  "state": "authenticated",
  "common_name": "Anonymous",
  "username": "Anonymous",
  "status": "authenticated"
}

One comment here - I would prefer if you change property "private" to something "isPrivate" or whatever you pick up. This is keyword in Royale. It is probably overreaction but we may have problems in minification with this.

piotrzarzycki21 commented 10 months ago

@JoelProminic I have made UI changes and used your API - we should be good and ready for first tests. I have merged along with your agent changes everything to main.

piotrzarzycki21 commented 10 months ago

I have just pushed last updates before the meeting. Everything should be ready. We can discuss what should be improved more.

JoelProminic commented 10 months ago

Some notes from the meeting:

In the Edit form, we should display a label for the password rather than showing a disabled field:

private="true": "Password set"
else: "No password required"

The "Change" button should be available in both cases, and this should make the text input appear for the password. Currently the Change button doesn't do anything.

If the user does not click the "Change" button, then don't pass the password parameter to GenesisDiretoryUpdate. The Update agents are designed to not change a property if no parameter is passed, but it will set the value to "" if the parameter is passed with an empty value (which will clear any existing password).

In the datagrid, we should add a new column to show the public/private status. For example:

"Password Required" column with checkbox/icons - I think I like this solution the best
"Type" column with "Public" or "Private" values/icons

Please add a Refresh button above the datagrid - I forgot to include this in the demo.

In Genesis App, the "App details" button should be disabled if no value is set for DetailsURL

Label updates:

Change "Directory" column in Genesis App to "Directory Catalog"
Change "Genesis Directories" header in Additional directories to "Genesis Directories for Private Catalogs"
In Add form, change "Add Directory" to "Add Genesis Directory"
In Edit form, change "Edit Directory" to "Edit Genesis Directory"
Change "Genesis App" to "Genesis Apps" in the sidebar (this is a bit off topic)

JoelProminic commented 10 months ago

@piotrzarzycki21, pushed an update for the GenesisRead agent. There were no Form/View updates, so you can just use Project > Run on Vagrant on the _Agents project to update the agents.

Currently this is failing for local Genesis Directory instances. I sent you a link with a remote address for testing.

I don't have updates for GenesisInstall yet. Please make sure to add the directory parameter to GenesisInstall so that I can test tomorrow once I get the agent updated.

piotrzarzycki21 commented 10 months ago

Add Genesis Directory

@JoelProminic I have addressed all issues from above list. Let's check over the call.

piotrzarzycki21 commented 10 months ago

I don't have updates for GenesisInstall yet. Please make sure to add the directory parameter to GenesisInstall so that I can test tomorrow once I get the agent updated.

I have added directory to installation process.

JoelProminic commented 10 months ago

I went through the issues reported here, and they are mostly fixed. A couple remaining issues:

I can't clear a password by clicking the Change button and leaving the password blank - the password parameter is not passed in this case.
I don't have an example to test for "App details" yet - the remote directory uses the original application links. I'll test this when I can.

An issue for my side:

If an installed application matches applications from multiple repositories, then the current GenesisRead code creates multiple entries under Installed Apps. I'm not sure how we can easily determine which repository was used to install an app - perhaps I can check the version?

piotrzarzycki21 commented 10 months ago

I can't clear a password by clicking the Change button and leaving the password blank - the password parameter is not passed in this case.

@JoelProminic I just pushed fix for this.

JoelProminic commented 10 months ago

I implemented the updates for additional Genesis Directories in GenesisInstall. I added a view to the database, so I'll need to send out a new empty database.

I also tested "private" repositories. I found that private repositories did not require the password for the REST API, only for the console command. The sensitive data and files should be protected, so I'll discuss whether this matters with @dpastov and @JustinProminic.

I am currently "escaping" the url and password for the console command by wrapping them in double-quotes.

TODO:

Automatically add a GenesisDirectory entry when installing genesis-directory.

piotrzarzycki21 commented 10 months ago

@JoelProminic after deploying newest database and agents I have following errors when I'm trying to get Genesis Apps and Additional dirs:

piotrzarzycki21 commented 10 months ago

I have found issue with behaviour of error popup window in Additonal directories - it's fixed, so pickup the next build.

JoelProminic commented 10 months ago

@piotrzarzycki21, the "Additional directories" issue seems to be because the new view is missing. Did you deploy the database I sent yesterday? If you open the database, you should see:

I did a fresh test by deploying the database I sent yesterday, and this seems to work fine for me.

piotrzarzycki21 commented 10 months ago

@piotrzarzycki21, the "Additional directories" issue seems to be because the new view is missing. Did you deploy the database I sent yesterday? If you open the database, you should see:

I did a fresh test by deploying the database I sent yesterday, and this seems to work fine for me.

@JoelProminic false alarm. I redeployed everything again and all is working now. Thanks.

JoelProminic commented 10 months ago

@piotrzarzycki21 thanks for confirming. I'm glad this was the cause.

I just updated GenesisInstall with the code to automatically create a directory entry when installing the genesis-directory app. This issue should be mostly ready for a release. I'll work on some final testing.

JoelProminic commented 10 months ago

This has worked well in my tests, so it should be ready for a release. I started on some release preparations, but I got stuck today on unrelated problems on the SHI/demo-tasks side.

JoelProminic commented 10 months ago

This is released with v0.1.8

Moonshine-IDE / Super.Human.Portal