Interface to Browse the Domino Server Databases

[JoelProminic]

We would like to add an interface that will provide a better way for the user to browse the databases in the Domino Directory.

This should provide options to view the databases as a list or by icons.

@dpastov and I will provide an agent based on the DbDirectory API

TODO: fill in more details

[JoelProminic]

@dpastov sent an example of the API. I won't include the original example here since it is for a Prominic server.

For this UI, the user will need to enter a server name that the Domino server has access to. This should default to the local server.

I'll need to create a new Java agent that uses @dpastov's code, which will take the server name as a parameter. UPDATE: To clarify, I expect this agent to be in SuperHumanPortal.nsf. It will only have access to other servers that the Server Admin has access to. For standalone SuperHumanImporter cases, I would generally expect the user to only have access to the same server.

The data will look something like this:

[
  {
    "title": "Base Level Example",
    "filePath": "BaseLevel.nsf",
    "replica": "...",
    "size": 6029312
  },
  {
    "title": "Multi-Level Example",
    "filePath": "A55XXX/nested/MultiLevelExample.nsf",
    "replica": "...",
    "size": 3921412096
  },
  ...

We will then want to organize these entries into an interface resembling a file browser. @JustinProminic suggested that we show the last two levels of nesting only using an interface similar to what we recently worked on for (SH-Meeting-Minutes](https://github.com/prominic/SH-Meeting-Minutes/issues/10) (Private). Note that there is not a lot of information to display for the databases currently, so we may wan to consider a more compact interface.

And then from this interface, the user will be able to select a database and automatically create a Database Bookmark in Custom Bookmarks. We'll need to show a simple form that will allow the user to select a category.

[piotrzarzycki21]

-- Bookmarks -- Browse My Server

[JoelProminic]

Here is the LotusScript code used by @dpastov's agent:

Option Public
Option Declare

Sub Initialize
    On Error GoTo errh

    Dim session As NotesSession
    Dim dbDir As NotesDbDirectory
    Dim db As NotesDatabase
    Dim jsonNav As NotesJSONNavigator
    Dim jsonObj As NotesJSONObject

    Set session = New notessession
    Set dbDir = New NotesDbDirectory("")

    Set jsonNav = session.Createjsonnavigator("[]")

    ' Loop through all databases on the server
    Set db = dbDir.GetFirstDatabase(DATABASE)
    While Not db Is Nothing
        Set jsonObj = jsonNav.appendObject()

        Call jsonObj.Appendelement(db.Title, "title")
        Call jsonObj.Appendelement(db.FilePath, "filePath")
        Call jsonObj.Appendelement(db.Replicaid, "replica")
        Call jsonObj.Appendelement(db.Size, "size")

        ' Get the next database
        Set db = dbDir.GetNextDatabase
    Wend

    ' Print or use the JSON string as needed
    Print "Status: 200"
    Print "Content-Type: application/json"
    Print jsonNav.stringify

    Exit sub
errh:
    Print "Status: 500"
    Print "Content-Type: text/html"
    Print "<h1>Server Error</h1><p>" & Error$ & " (" & CStr(erl) & ")</p>"
End Sub

I should be able to quickly do some thing similar for Java.

@JustinProminic confirmed that we want to enable this lookup for the local sever only at this point. We could consider adding a parameter to lookup other servers later.

[JoelProminic]

A key feature for this interface will be integration with the custom bookmarks:

• Show which databases already have bookmarks. I will add a property to indicate this
• Quickly create bookmarks for the databases, populating the server and database automatically. The label can also be populated with the title.
• Quickly delete bookmarks, with a confirmation prompt. To support this, I will need to include a key for the bookmark document (which may also server as the flag to indicate that there is an existing bookmark.
  • Note that a database can have multiple associated bookmarks, with different views and/or labels. We'll need to think about how to handle this case.
  • UPDATE: After discussing the multiple associated bookmark issue, the Delete action is a lower priority for now
• UPDATE: we also need a way to open the displayed database in the Notes Client/Nomad from the database browsing interface. I could generate the link JSON for this.

I'll need to add a view to lookup bookmarks based on the server and database name.

[JoelProminic]

The new GUI for this should be displayed under the Bookmarks menu: Bookmarks > Browse My Server. This entry should always exist, so that the user can add more databases later. When the user opens the database, they should see a an interfaces showing the databases from the JSON agent, organized using the tab interface we have been working on for SH-Meeting-Minutes.

[JoelProminic]

Thinking about this further, I think it makes more sense to reuse the link format for the JSON:

• Populate Name with the database title
• add "hasBookmark" and "replicaID"
• Optionally add "bookmarkCount" and "bookmarks"

          {
            "name": "Oil Services Products",
            "type": "database",
            "url": "notes://demo.STARTcloud.com/Customer/Products.nsf"
            "nomadURL": "...TODO..."
            "server": "demo.STARTcloud.com/STARTcloud",
            "database": "Customer/Products.nsf",
            "view":"",
            "replicaID":"...",
            "hasBookmarks":true,
            "bookmarkCount":1,
            "bookmarks:[
                ... possible future feature ...
            ]
          }

[JoelProminic]

We made a mockup for this interface during the meeting:

[JoelProminic]

I spent some time on the agent today, but I'm running into some unexpected errors on the imported agent. I'll continue investigating this, and hopefully I'll have something ready tomorrow. (FYI, there is another urgent task that may distract me from this).

[piotrzarzycki21]

Thinking about this further, I think it makes more sense to reuse the link format for the JSON:

• Populate Name with the database title
• add "hasBookmark" and "replicaID"
• Optionally add "bookmarkCount" and "bookmarks"

          {
            "name": "Oil Services Products",
            "type": "database",
            "url": "notes://demo.STARTcloud.com/Customer/Products.nsf"
            "nomadURL": "...TODO..."
            "server": "demo.STARTcloud.com/STARTcloud",
            "database": "Customer/Products.nsf",
            "view":"",
            "replicaID":"...",
            "hasBookmarks":true,
            "bookmarkCount":1,
            "bookmarks:[
                ... possible future feature ...
            ]
          }

@JoelProminic Quick question - your agent will return above items and I don't actually bother what Dmytro API returns, because you simply call it and I'll get above array of objects ?

[JoelProminic]

Right, the agent I'm working on will use the format from this comment rather than the format from Dmytro's agent. I chose this format to match the format for links/bookmarks.

My agent is almost done, but I ran into trouble with DXL Importer in my tests. I was busy yesterday with another task, but I'll try to make time to debug my agent today.

[JoelProminic]

@piotrzarzycki21, I pushed an initial version of the agent so that you had something to get started. You will only need to deploy the agents with this database (later updates will include a database update). Incomplete components:

• url and nomadURL are not populated. I'm hoping I can reuse the old logic for this
• hasBookmark is always false for now. If you need to test this before I finish the update, you can change this line: https://github.com/Moonshine-IDE/Super.Human.Portal/blob/55ed341233c6e0d614705b5f93e56aefd7b32336/Super.Human.Portal_Agents/src/main/java/CustomBookmarkAgents/DatabaseRead.java#L77

FYI, I'm still getting DXL Importer errors on the Super.Human.Installer servers. If you start getting agent errors after the update, then revert to the last database copy I sent you and let me know. I didn't make any changes to the DXL importer scripts recently, so I don't know why I'm getting problems.

[piotrzarzycki21]

@JoelProminic after deploying your agents I'm getting following on application startup:

"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\n<html lang=\"en\">\n<head>\n<title>Server Login</title>\n<script language=\"JavaScript\" type=\"text/javascript\">\n<!-- \nfunction doError(){return true;};window.onerror=doError;\n// -->\n</script>\n</head>\n<body text=\"#000000\" onLoad=\"document.forms[0].username.focus(); document.forms[0].username.select();\" bgcolor=\"#FFFFFFFF\">\n\n<form method=\"post\" action=\"/names.nsf?Login\">\n<table border=\"0\" cellspacing=\"0\" cellpadding=\"0\" bgcolor=\"#FFFFCC\">\n<tr><td width=\"100\">&nbsp;</td><td><h1>Server Login</h1>Please type your user name and password<br>\n</td><td width=\"100\">&nbsp;</td></tr>\n\n<tr><td height=\"20\" colspan=\"3\">&nbsp;</td></tr>\n\n<tr><td width=\"100\" align=\"right\"><label for=\"user-id\">&nbsp;User name:&nbsp;&nbsp;</label></td><td>\n<input type=\"text\" size=\"20\" maxlength=\"256\" name=\"username\" id=\"user-id\"></td><td>&nbsp;</td></tr>\n\n<tr><td width=\"100\" align=\"right\"><label for=\"pw-id\">Password:&nbsp;&nbsp;</label></td><td>\n<input type=\"password\" size=\"20\" maxlength=\"256\" name=\"password\" id=\"pw-id\" autocomplete=\"off\">\n<input type=\"hidden\" name=\"redirectto\" value=\"/SuperHumanPortal.nsf/ConfigRead?OpenAgent\"></td><td>&nbsp;</td></tr>\n\n<tr><td height=\"20\" colspan=\"3\">&nbsp;</td></tr>\n\n<tr><td width=\"100\">&nbsp;</td><td>\n<input type=\"submit\" value=\"Log In\"></td><td>&nbsp;</td></tr>\n\n<tr><td height=\"20\" colspan=\"3\">&nbsp;</td></tr>\n</table>\n</form>\n</body>\n</html>\n"

VM1423:1 Uncaught SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
    at JSON.parse (<anonymous>)
    at model.proxy.login.ProxyLogin.model_proxy_login_ProxyLogin_onGeneralConfigurationLoaded (ProxyLogin.js:227:43)
    at org.apache.royale.events.EventDispatcher.fireListeners (EventDispatcher.js:113:23)
    at goog.events.EventTarget.dispatchEventInternal_ (eventtarget.js:381:26)
    at org.apache.royale.events.EventDispatcher.dispatchEvent (EventDispatcher.js:88:37)
    at org.apache.royale.core.ElementWrapper.dispatchEvent (ElementWrapper.js:223:76)
    at org.apache.royale.net.HTTPService.progressHandler (HTTPService.js:198:10)

[JoelProminic]

Sorry, I missed your update earlier. This looks like the same error I am getting when importing the agents. I don't know what has changed to cause this - I only see changes to the src code and the agentProperties configuration, and I confirmed that the old agents are failing as well.

[piotrzarzycki21]

Question is what could help you debug that stuff ? After deploying your agents released Super Human Portal stopped working as well.

[JoelProminic]

In the short term, I confirmed I could fix the problem like this:

1. Open the database in the Notes Client (File > Application > Open, select your server, SuperHumanPortal.nsf)
2. File > Application > Access Control...
3. Select the Anonymous entry and change the Access level from Depositor to Designer. Click OK to save
4. Hard-refresh Super.Human.Portal (or clear the cache). It should work now

However, this is not a proper permission level for Anonymous, so we need a better long-term solution. I found this fix because I have seen similar errors before, but I don't know why it suddenly started triggering again now.

[piotrzarzycki21]

Update what is ready on UI sight:

• Menu displaying parsed path

In the short term, I confirmed I could fix the problem like this:

1. Open the database in the Notes Client (File > Application > Open, select your server, SuperHumanPortal.nsf)
2. File > Application > Access Control...
3. Select the Anonymous entry and change the Access level from Depositor to Designer. Click OK to save
4. Hard-refresh Super.Human.Portal (or clear the cache). It should work now

However, this is not a proper permission level for Anonymous, so we need a better long-term solution. I found this fix because I have seen similar errors before, but I don't know why it suddenly started triggering again now.

Your suggestions worked! I will try to use data which I got and see if my algorithm for parsing working tomorrow.

[JoelProminic]

Some notes from the meeting today:

• [x] Scroll is not working for sidebar control. We discussed whether it would make sense to have separate scroll bars for the two columns
• [x] Sorting
  • Folders should be displayed at the top
  • Otherwise, the entries should be sorted alphabetically
• [x] Make it more clear which entries are folders.
  • Add an icon?
  • Add a "/" after the label?
• [x] What to display before a database is selected
  • summary of features for this interface
  • alternative iterface for displaying the databases/folders in this directory
• [x] Change how current selection is displayed at top of browse bar
  • Try prefixing with "Selection: "
• [x] We would like the action buttons for the database to be in more prominent location. Currently, they feel lost at the bottom with the Copy to Clipboard buttons
  • Updating order: Open in Client | Open in Nomad | Add Bookmark
• [x] Copy to Clipboard
  • change to icon
  • move closer to text - we use more horizontal space for this interface compared to the bookmarks interface.
• [ ] Future feature: Hide default templates, or the templates in general (.ntf)
• [x] Refresh button. This will probably be rarely used, but we would like to have it if needed.

[JoelProminic]

Example text for the main panel when no database is selected:

Select a database or template in the sidebar to see more details and options.

Available features:

• Open the database
• Quickly create bookmarks for the database.

[piotrzarzycki21]

I have updated your comment to have check marks and treat all of that as a tasks.

[piotrzarzycki21]

I have merged changes from my branch to main - I will be working on main from now on.

[piotrzarzycki21]

@JoelProminic I have pushed some update to interface - it's partially ready for review. I will be working on this farther tomorrow.

[JoelProminic]

I pushed an agent update to populate url and nomadURL based on the same code used for bookmarks. You should only need to update the agents to get this.

The remaining agent update is implement hasBookmarks, bookmarkCount and bookmarks. These will require a new view. (I also needed to add a new view for a pending change for bookmark sorting, so I will probably do that at the same time.)

[piotrzarzycki21]

Thanks! What do you mean by Bookmark sorting ? Are you going to send me bookmarks sorted in some way ?

[JoelProminic]

When I was working on converting the Prominic Portal, I found that the sorting didn't look like I expected. I don't remember the exact details for this, but I remember I fixed this by creating a view called Bookmarks/FlatForAgent in the Prominic database. The new view was sorted by group, then name. We may be better off updating the sorting logic on the Royale side, but I'll need to review what was wrong for the portal tests.

[piotrzarzycki21]

Some notes from the meeting today:

• [x] Scroll is not working for sidebar control. We discussed whether it would make sense to have separate scroll bars for the two columns

• [x] Sorting

  • Folders should be displayed at the top
  • Otherwise, the entries should be sorted alphabetically

• [x] Make it more clear which entries are folders.

  • Add an icon?
  • Add a "/" after the label?

• [x] What to display before a database is selected

  • summary of features for this interface
  • alternative iterface for displaying the databases/folders in this directory

• [x] Change how current selection is displayed at top of browse bar

  • Try prefixing with "Selection: "

• [x] We would like the action buttons for the database to be in more prominent location. Currently, they feel lost at the bottom with the Copy to Clipboard buttons

  • Updating order: Open in Client | Open in Nomad | Add Bookmark

• [x] Copy to Clipboard

  • change to icon
  • move closer to text - we use more horizontal space for this interface compared to the bookmarks interface.
• [ ] Future feature: Hide default templates, or the templates in general (.ntf)
• [x] Refresh button. This will probably be rarely used, but we would like to have it if needed.

Apart of Future feature - all issues has been addressed.

[JoelProminic]

We reviewed the file browser interface with Doug today, and he requested some changes.

• [x] Use pure black text for the folders - the current grey text doesn't have enough contrast
• [ x] Add keybindings for the Next and Previous buttons to the left and right arrow keys. -> #47
• [x] Change the hamburger icon to a right or left arrow, to indicate expand or collapse. Hamburger menus usually indicate a dropdown
• [x] If a level of nesting is hidden in the sidebar, show a thin column with arrows or ... to indicate a hidden columns. Here is how it looks in macOS Finder:

[piotrzarzycki21]

@JoelProminic I have made couple of improvements before today's call. I did also fix issue where clicking on "Menu" in breadcrump didn't refresh properly menu items. I did not fix yet issue which you have discovered during call with deeply nested elements, but my changes may influence that scenario - something which I need to check.

[piotrzarzycki21]

• [x] Remove feature indicating parent folder
• [x] Gray out buttons next/prev when we are last/first item -> #48
• [x] Move expand/collapse button to the top blue bar - add double arrows icons
• [x] Sorting folders and move to the top in deep nestead elements

[piotrzarzycki21]

I have splited remaining bugs/enhancements to separate issues #47 and #48. I'm closing this one.

[JoelProminic]

I'm reopening this issue so that I remember to implement the agent-side logic for hasBookmarks.

[JoelProminic]

Test	Found bookmarks	Notes
Create a bookmark from the Browse My Server interface	Yes
Create a bookmark manually that matches a database	Yes
No bookmarks for database	No	I compared my updated JSON to the full output of the old agent version, and did not see any mismatches until I started adding bookmarks
Multiple matching bookmarks	Yes
Case mismatch	Yes
Server includes /ORG	Yes
Server does not include /ORG	Yes	The Browse My Server Add button includes the organization, but a manually entered database may not (and it is not required).
Leading character on server name`	No
Trailing character on database name	No
Leading and trailing whitespace on server	Yes
Leading and trailing whitespace on database	Yes

For the whitespace cases, I made a small update to the link code to trim the database string when building the Notes and Nomad URLs.