Superuser management tools

[amplifi]

(Updating initial description with the requirements and feedback collected)

Problem Statement

Currently both staging and demo environments have many internal test projects that need to be cleaned, as they are impacting on the maintenance of those environments due to performance, stability, and potential migrations needed.

We need a minimum set of super user tools to manage this, to be able to remove projects from either staging or demo.

Context / Use Case

When testing new features, or working with partners on early versions of questionnaires, we are generating a lot of projects that have no value after this testing and then should be deleted.

However, there may be other projects that are still used in trainings with partners, when working on intermediate versions of questionnaires, or when testing a recently deployed feature. So we would also need a feature to prevent those projects being deleted.

User Stories

• US1: As a super user, I want to tag existing projects/organizations in the platform as "to be retained", so when doing a general clean-up of the environment all the projects will be removed aside from those tagged as "to be retained"
• US2: As a super user, I want to select the projects/organizations to be removed
• US3: As a super user, I want to select the projects/organizations to be archived

Description

1. In the project/organization index, the super user will be able to batch select projects or organizations using check boxes on the left. Then the super user will be able to:

• Retain those projects
• Archive those projects
• Delete those projects

2. A confirmation modal should appear before actually committing the action.

3. If a project in an organization is marked for retention, the corresponding org should be marked for retention automatically. Similarly, if all the projects in an org are marked for deletion, then the org should be marked automatically.

4. There should be a badge indicating that a project (or an org) is marked as retained.

5. Deletion of "non-retained" projects and orgs will be done as part of the release process.

6. Should not be possible to delete an organization that has projects marked as retained.

----ORIGINAL---- Currently, there's no functionality for centralized management of projects or organizations, and staging and demo environments are cluttered with (mostly internal) test projects. This negatively impacts performance, stability, security, complexity of future development & migrations, and public-facing site aesthetics. It also leaves us with no option but manual intervention at the database level when such issues occur. We need a basic set of superuser tools to be able to manage and maintain the platform environments.

When a superuser is logged in, the minimum toolset should include:

• Batch archive projects. The Projects page should include a column of checkboxes and an "Archive" button, allowing one or more projects to be archived at a time by the superuser.
• Batch archive organizations. The Organizations page should include a column of checkboxes and an "Archive" button, allowing one or more organizations to be archived at a time by the superuser. Archiving an organization should archive all projects under that organization.
• The Users page should include links in the Organizations column instead of plaintext names.
• When sorting by Last Login on the Users page, the resulting display should be chronological (bug; it displays out of order).

We should consider also including batch delete options in addition to batch archive for both projects and organizations, which would require additional work as platform data isn't deleted under present models.

Items for an expanded superuser toolset are covered under #983.

[amplifi]

I also like @SteadyCadence's idea to add a message/banner on demo letting users know orgs and projects are subject to pruning in that env. It'd be a good heads up even if we're not doing any kind of scheduled refresh.

[wonderchook]

So the issue with refreshing on the demo environment is we need to think about how we are going to better manage users who we don't have contact with. How do we make sure people don't do "real" things on the demo environment? Should there be 2 environments in the first place, are there features that would make this all easier.

[amplifi]

@wonderchook The superuser tools I'm proposing for batch archiving and deletion would eliminate the need for demo env refreshes. I was including @SteadyCadence's suggestion of messaging because I think it's important to notify casual users up front that their projects may not be retained, so that we're able to archive/delete as warranted.

[wonderchook]

@amplifi okay, was only concerned about the not retaining them. The tools all read as the right thing to me.

I think eventually we might want to consider a way to import projects across servers. I think that might be more if/when we have multiple installations including on users networks.

[amplifi]

@wonderchook Are there use cases that wouldn't be covered by a user's ability to export a project then import it on another platform instance? Enabling any private inter-environment routes would breach our isolation of environments, and I'm not sure we should be routing project data over public internet once it's in the platform (if the user chooses to do so via export/import, I think that exposure is more transparent to them).

[dpalomino]

@wonderchook Are there use cases that wouldn't be covered by a user's ability to export a project then import it on another platform instance? Enabling any private inter-environment routes would breach our isolation of environments, and I'm not sure we should be routing project data over public internet once it's in the platform (if the user chooses to do so via export/import, I think that exposure is more transparent to them).

At least for now I think with the export/import functionality we should be fine for moving projects from demo to platform (it is not a very common scenario). It would be nice though to have a workaround to download the xlsform used in the original project (now when the project already contains some data it's not possible to download it):

[amplifi]

@dpalomino The xlsform download sounds like it should be a user-facing option. It'd be good to have a separate issue for tracking that.

[dpalomino]

ok, thanks @amplifi. I will open it now.

[clash99]

THESE TWO WIREFRAMES ARE OUTDATED - SEE FINAL SET OF WIREFRAMES DATED 07/13/17

Here are two options for bulk archiving of projects by a super user. The first option is more scalable since it also allows for the additional delete option. Initially only the archive action would be available in the dropdown. The second mock only allows for archiving (not truly bulk) but would be quicker development time (I believe).

[wonderchook]

I'm a little afraid of the 1st one, but I think it is what we need.

What if the archive button was added next to the projects like the 2nd and then the 1st was delete only where you could select multiple? Should we require reauthentication for deleting?

[clash99]

For the first option, I think there should be a confirmation modal for both archive and delete actions. I was even wondering if a project should have to be archived to be eligible to be deleted. Or does that extra step make sense? The case would be to avoid deleting projects that really should just be archived. The flow would be to select "Archived" from the top display dropdown, check the box, select "Delete" from the bottom action dropdown, and then submit. Thoughts?

[wonderchook]

I was thinking a project has to be archived to be deleted too. I think we want to make it pretty hard to delete things.

[dpalomino]

Hey @oliverroick. We were talking about implementing in next sprint:

• The ability of tagging projects as "un-deletable", so we can skip the ones used by partners or for demos in the programs team
• Ability to remove the other projects in demo

Maybe if you have time for this after #1476?

Thanks!

[oliverroick]

Before I can implement this, we should agree on design and approach for these features.

The ability of tagging projects as "un-deletable", so we can skip the ones used by partners or for demos in the programs team

• Is it a flag we set on the project model.
• Where does the UI sit for this?
• Who can mark projects as undeletable? Superuser only?

Ability to remove the other projects in demo

• What are "other projects"?
• What does the UI look like?
• What does delete mean? Actually erasing the project and all its data from the database?
• Who can delete projects? Superusers only?

We should apply the processes that we agreed upon in Madrid. Which means we should write a specification to outline what these features will look like.

[dpalomino]

We should apply the processes that we agreed upon in Madrid. Which means we should write a specification to outline what these features will look like.

Thanks @oliverroick. Yes, agreed, as there was a proposal ongoing for the UI (@clash99's comment) I missed this one. I'm working on it today.

[dpalomino]

Hey @oliverroick, @amplifi, @clash99

Can you please take a look and confirm if this properly reflects what we've been discussing? When I have your go I'll create the DR. Happy to discuss in a call as well.

Thanks!

Problem Statement

Currently both staging and demo environments have many internal test projects that need to be cleaned, as they are impacting on the maintenance of those environments due to performance, stability and potential migrations needed.

We need a minimum set of super user tools to manage this, to be able to remove projects from either staging or demo.

Context / Use case

When testing new features, or working with partners on early versions of questionnaires, we are generating a lot of projects that have no value after this testing and then should be deleted.

However, there may be other projects that are still used in trainings with partners, when working on intermediate versions of questionnaires, or when testing a recently deployed feature. So we would also need a feature to skip those projects of being deleted.

User Stories

• US1: As a super, I want to tag existing projects in the platform as "un-deletable", so when a do a general clean-up of the environment all the projects but those tagged as "un-deletable" will be removed
• US2: As a super user, I want to individually select the projects to be removed from the platform
• US3: As a super user, I want to be able to remove all projects in the platform older than 30 days but skipping those tagged as "un-deletable"

Description

User Story 1 - Tagging un-deletable projects

For this feature, I would suggest the UI proposed by @clash99 (see this wireframe), having checkboxes for each project and a drop-down menu including the "Mark as un-deletable" option, and then a submit button.

User Story 2 - Individually remove projects

Similarly we can use the same UI with the drop-down menu including a "Delete" option, and then a submit button.

For this case we should have a confirmation modal.

User Story 3 - Remove all projects skipping "un-deletable" ones

I would suggest not to have a UI for this, but just periodically execute a script for this.

I think we could have two options here:

1. Use the project creation date for filtering the projects to be deleted
2. Use the project timestamp

If it is possible and the effort is similar, I think (2) would be preferable, so we are by default skipping those projects that are "active" (i.e. being edited or with new records)

[clash99]

@dpalomino - I'm a little concerned about the undeletable option. Couple of questions/comments:

• Is the person creating the project for the purpose of a demo a superuser? If not then do they communicate with a superuser to mark it as undeletable? I'm confused by the work flow.

• I don't like the idea of adding it to the checkbox/ dropdown combo wireframe. I would suggest adding it to the project details page (similar to the public/private toggle) and then when you see it in the index, those projects have a disabled/ no checkbox so their status can't be changed. (This is assuming these projects are more rare so that setting this option one project at a time isn't too time consuming.)

• I also hate the word undeletable but I can't think of a better one and I guess it is clear what it means! :)

[dpalomino]

Thanks @clash99!

Is the person creating the project for the purpose of a demo a superuser? If not then do they communicate with a superuser to mark it as undeletable? I'm confused by the work flow.

I think this is only going to impact to a few bunch of projects, those used for demoing (the ones in the Programs organization) and the ones currently being used for training/testing purposes with partners (not more than 5-6 at a time I think), and either @SteadyCadence or me will be involved, so it is just communicating to a super user when one of this projects need to be marked as undeletable (my estimation is that this is not going to happen more than once a week or so).

I don't like the idea of adding it to the checkbox/ dropdown combo wireframe. I would suggest adding it to the project details page (similar to the public/private toggle) and then when you see it in the index, those projects have a disabled/ no checkbox so their status can't be changed. (This is assuming these projects are more rare so that setting this option one project at a time isn't too time consuming.)

Sure, I was just suggesting this for reusing the wireframe you created. As this is only affecting to super users the UX is not critical, but if you want to include in the project details I agree that this is a better place if that is not adding more work.

I also hate the word undeletable but I can't think of a better one and I guess it is clear what it means! :)

haha! Agree :-) I'm open to suggestions, maybe the "indestructible" projects? :-P

[clash99]

@dpalomino - The ideal workflow for making these demo projects stick around would be for the system to know you are Cadasta internal staff and provide you with the option of setting it "official/undeletable" when you create it. I don't think we have that mechanism though. Do we?

[dpalomino]

Yes, that would be ideal of course @clash99. But I guess that would mean to add a new permission to the system "Cadasta Staff" to be something intermediate between the super user and a regular user? Would that be interfering with the tutelary work that is ongoing? Or can we whitelist a set of Cadasta users just for this?

Thoughts @bjohare? Do you think this would interfere in the (get-rid-of) tutelary work?

[amplifi]

A few quick items:

1) We already have a field in the user accounts table for is_staff. We are missing a UI to enable a superuser to mark other user accounts as Cadasta staff.

2) It should be a superuser marking demo projects to be saved/excluded from deletion by request only, not an end user or staff member. It shouldn't be easy to retain projects because we want the demo environment model to be impermanent by default. There should be very few exceptions to this rule. The workflow would be to create a GitHub Issue to request that a project be excluded from deletion, assigned to a superuser.

3) Superusers should be able to view a checkbox per project and per org on the project and org pages, respectively, to enable batch selections with a single save button (like Option 1 wireframe above). If a project is marked for exclusion from deletion, the corresponding org should be marked automatically. Similarly, if an org has only one project, and the project is marked for deletion, the org should be marked automatically.

4) There should be a badge or similar icon visible to all users to indicate on the orgs list, projects list, and project overview page that it has been marked for exclusion.

[amplifi]

@dpalomino Re: user stories:

We can call it "Retain." Retain/Archive/Delete in the drop-down.

There should be a confirmation modal for each action (not just US2), showing the action and the selected items.

For US3, we will not be deleting projects via script, nor will we be running random scripts against live production databases. Deletion of obsolete demo projects should be performed as part of the release process. Eligibility for deletion should be determined based on project creation date, not when it was last updated -- there should be no need to use a demo project more than 30 days without it being requested for retention.

[dpalomino]

Thanks @amplifi

We already have a field in the user accounts table for is_staff. We are missing a UI to enable a superuser to mark other user accounts as Cadasta staff.

Ah, thanks for letting us know.

It should be a superuser marking demo projects to be saved/excluded from deletion by request only, not an end user or staff member. It shouldn't be easy to retain projects because we want the demo environment model to be impermanent by default. There should be very few exceptions to this rule. The workflow would be to create a GitHub Issue to request that a project be excluded from deletion, assigned to a superuser.

That's fine for me. I was suggesting to avoid overloading the superusers, but it's going to be really few projects in this list. I can update the requirements to reflect this.

Superusers should be able to view a checkbox per project and per org on the project and org pages, respectively, to enable batch selections with a single save button (like Option 1 wireframe above).

Ok, but I think it's going to be very few projects, so I think batch selections are not going to be critical. But if we are not going to use the is_staff field, it's true that the batch selection makes more sense here (unless it'd be much more complicated to implement, but I don't think it'd be the case).

If a project is marked for exclusion from deletion, the corresponding org should be marked automatically. Similarly, if an org has only one project, and the project is marked for deletion, the org should be marked automatically.

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

There should be a badge or similar icon visible to all users to indicate on the orgs list, projects list, and project overview page that it has been marked for exclusion.

Good point, thanks! And I'd say also for the organization overview page. I can include this in the requirements (similar badge to the private or archived ones).

We can call it "Retain." Retain/Archive/Delete in the drop-down.

Retain, great. That sounds good ;-)

[amplifi]

@dpalomino

Ok, but I think it's going to be very few projects, so I think batch selections are not going to be critical. But if we are not going to use the is_staff field, it's true that the batch selection makes more sense here (unless it'd be much more complicated to implement, but I don't think it'd be the case).

The purpose behind the batch/checklist format isn't just to simplify a large volume of operations. It provides a single, unified interface for managing projects and organizations. It prevents superusers from having to load project data (by loading a project's dashboard page) just to retain/archive/delete a project. It reduces staff time spent on these requests by shrinking the superuser workflow from 1) find a project in the list, click on a project link, load the project data, mark it for retention, confirm the action on the modal, return to the project page, repeat for each project and org... to 2) load the project page, tick the box(es) for the desired project(s), select the action from a drop-down, confirm the modal. It minimizes context-switching for the superuser. The single batch interface for retain/archive/delete is less to QA, has fewer potential points of failure to maintain going forward, and is actually less work to implement.

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

No; this is essential. It's trivial to add these conditional requirements and reduces the risk of data corruption. If someone were to mark a project for retention, but not mark the organization, we could end up in a situation where an org has been deleted but its project has not. (Of course, the deletion functionality should prevent deletion of orgs with existing projects, but we shouldn't only be putting this critical check in one place. We should be requiring data consistency across the platform.)

Good point, thanks! And I'd say also for the organization overview page. I can include this in the requirements (similar badge to the private or archived ones).

Yes, I included the organizations list page in my comment above.

[clash99]

@dpalomino Might be dumb questions but trying to think through the logic scenarios and want to be sure I understand:

• If a project is marked as deleted, does it disappear from the list immediately (after confirmation)?
• Are archived projects able to be marked to retain? If so, does that change their status to active?
• Does a project or organization have to be archived first to be deleted?
• What happens if a user tries to archive or delete an organization that contains a retained project?

[clash99]

Wireframes

• PDF containing all wireframes

Project Index

Organization Index

Warning Messages for creating new org or project

These don't address issues for management of the user area (#983). I wasn't sure if that was part of this project but I will add those wireframes in that issue.

[amplifi]

@clash99

If a project is marked as deleted, does it disappear from the list immediately (after confirmation)?

A project should disappear from the list when it has been deleted. Confirming any operation (archive, retain, delete) on this page should refresh the view. When the project is deleted, it will automatically no longer appear in the project list. Similar to how when a project is archived, clicking to confirm the operation refreshes the view and the archived project has the corresponding badge next to it.

Are archived projects able to be marked to retain? If so, does that change their status to active?

Yes, archived projects can be marked for retention. No, it shouldn't change their status to active; they should remain archived. The importance of keeping "Archived" as a distinct state comes into play in the next question, as well.

Does a project or organization have to be archived first to be deleted?

No, this shouldn't be required. "Archived" and "able to be deleted" should remain distinct states, because they could mean very different things across environments. In production, there's no expectation of deletion, so archiving implies its own context.

What happens if a user tries to archive or delete an organization that contains a retained project?

This should never be possible. Item #3 in my comment above would preclude it: If a project is marked for exclusion from deletion, the corresponding org should be marked automatically.

[amplifi]

@clash Wireframes look great (as always)! To address the questions from them:

• Yes, the dropbox should only apply to the current page.
• Adding a submit button for the dropbox is a good idea -- barrier to selecting the wrong option.
• I also like the idea to add the last_updated column to the org list so it mirrors the project list.

Re: wireframe pg. 1 no. 7, the dialogue for the retention modal should use a different word than "Active", as archived projects can also be marked to retain.

Re: pg. 3 no. 1, I'd use the number "30" and bold the clause "all projects will be removed after 30 days."

Otherwise, 👍

[dpalomino]

Thanks @amplifi for the feedback!

The purpose behind the batch/checklist format isn't just to simplify a large volume of operations. It provides a single, unified interface for managing projects and organizations. It prevents superusers from having to load project data (by loading a project's dashboard page) just to retain/archive/delete a project. It reduces staff time spent on these requests by shrinking the superuser workflow from 1) find a project in the list, click on a project link, load the project data, mark it for retention, confirm the action on the modal, return to the project page, repeat for each project and org... to 2) load the project page, tick the box(es) for the desired project(s), select the action from a drop-down, confirm the modal. It minimizes context-switching for the superuser. The single batch interface for retain/archive/delete is less to QA, has fewer potential points of failure to maintain going forward, and is actually less work to implement.

Ok, if it's even easier to implement then I think we all agree...

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

No; this is essential. It's trivial to add these conditional requirements and reduces the risk of data corruption. If someone were to mark a project for retention, but not mark the organization, we could end up in a situation where an org has been deleted but its project has not. (Of course, the deletion functionality should prevent deletion of orgs with existing projects, but we shouldn't only be putting this critical check in one place. We should be requiring data consistency across the platform.)

Ok, if this is not adding much complexity to the implementation I think it'd be useful.

And thanks @clash99! Answering the questions:

If a project is marked as deleted, does it disappear from the list immediately (after confirmation)?

Yes.

Are archived projects able to be marked to retain? If so, does that change their status to active?

Yes.

Does a project or organization have to be archived first to be deleted?

No, I think that's not necessary.

What happens if a user tries to archive or delete an organization that contains a retained project?

It'd not be allowed. The organization cannot be deleted when there are projects within the organization. And the organization should not be archived when there are active projects within the organization.

I think a simple error message for this would be sufficient.

[dpalomino]

Thanks @clash99 for the wireframes! Looks really great :-)

• I agree with the submit button, I think it'd be good to include it.
• Regarding the "active" word for the retain modal, what about: "You have selected to retain the following projects, meaning that they will not be removed after 30 days after their creation until the status is manually changed. Are..."

[clash99]

Thanks @amplifi and @dpalomino for the feedback. The 7/13/17 wireframes above have been updated. Changes include:

• Added "Go" button behind selection dropdown on index pages
• Updated text in retain confirmation modal to not include "active"
• Added example of error message in modal
• Added last updated column in organization index
• Updated warning messages (I also remove the contact us link on this - should we provide some sort of countdown? perhaps this is overkill for now)
• Updated text to supply more details on all pages

Let me know if you see anything else that needs updating.

[oliverroick]

Ok, so this discussion got a bit out of hand. This needs a formal decision record, including a functional and technical spec. Otherwise, development will be a big, untestable mess.

I'll lead on creating the decision record; we're aiming to finish and approve the DR for this sprint, development will have to wait for the next sprint.

[dpalomino]

Hey @oliverroick, @clash99, @amplifi

I've updated the description with the final version of the requirements including all the feedback (thanks!) received. Please let me know if I'm missing something.

I'll lead on creating the decision record; we're aiming to finish and approve the DR for this sprint, development will have to wait for the next sprint.

Yes, that sounds reasonable to me... thanks @oliverroick!