Cadasta / cadasta-platform

[DEPRECATED] Main repository of the Cadasta platform. Technology to help communities document their land rights around the world.
https://demo.cadasta.org
GNU Affero General Public License v3.0
53 stars 81 forks source link

Superuser management tools #1418

Open amplifi opened 7 years ago

amplifi commented 7 years ago

(Updating initial description with the requirements and feedback collected)

Problem Statement

Currently both staging and demo environments have many internal test projects that need to be cleaned, as they are impacting on the maintenance of those environments due to performance, stability, and potential migrations needed.

We need a minimum set of super user tools to manage this, to be able to remove projects from either staging or demo.

Context / Use Case

When testing new features, or working with partners on early versions of questionnaires, we are generating a lot of projects that have no value after this testing and then should be deleted.

However, there may be other projects that are still used in trainings with partners, when working on intermediate versions of questionnaires, or when testing a recently deployed feature. So we would also need a feature to prevent those projects being deleted.

User Stories

Description

  1. In the project/organization index, the super user will be able to batch select projects or organizations using check boxes on the left. Then the super user will be able to:
  1. A confirmation modal should appear before actually committing the action.

  2. If a project in an organization is marked for retention, the corresponding org should be marked for retention automatically. Similarly, if all the projects in an org are marked for deletion, then the org should be marked automatically.

  3. There should be a badge indicating that a project (or an org) is marked as retained.

  4. Deletion of "non-retained" projects and orgs will be done as part of the release process.

  5. Should not be possible to delete an organization that has projects marked as retained.

----ORIGINAL---- Currently, there's no functionality for centralized management of projects or organizations, and staging and demo environments are cluttered with (mostly internal) test projects. This negatively impacts performance, stability, security, complexity of future development & migrations, and public-facing site aesthetics. It also leaves us with no option but manual intervention at the database level when such issues occur. We need a basic set of superuser tools to be able to manage and maintain the platform environments.

When a superuser is logged in, the minimum toolset should include:

We should consider also including batch delete options in addition to batch archive for both projects and organizations, which would require additional work as platform data isn't deleted under present models.

Items for an expanded superuser toolset are covered under #983.

amplifi commented 7 years ago

I also like @SteadyCadence's idea to add a message/banner on demo letting users know orgs and projects are subject to pruning in that env. It'd be a good heads up even if we're not doing any kind of scheduled refresh.

wonderchook commented 7 years ago

So the issue with refreshing on the demo environment is we need to think about how we are going to better manage users who we don't have contact with. How do we make sure people don't do "real" things on the demo environment? Should there be 2 environments in the first place, are there features that would make this all easier.

amplifi commented 7 years ago

@wonderchook The superuser tools I'm proposing for batch archiving and deletion would eliminate the need for demo env refreshes. I was including @SteadyCadence's suggestion of messaging because I think it's important to notify casual users up front that their projects may not be retained, so that we're able to archive/delete as warranted.

wonderchook commented 7 years ago

@amplifi okay, was only concerned about the not retaining them. The tools all read as the right thing to me.

I think eventually we might want to consider a way to import projects across servers. I think that might be more if/when we have multiple installations including on users networks.

amplifi commented 7 years ago

@wonderchook Are there use cases that wouldn't be covered by a user's ability to export a project then import it on another platform instance? Enabling any private inter-environment routes would breach our isolation of environments, and I'm not sure we should be routing project data over public internet once it's in the platform (if the user chooses to do so via export/import, I think that exposure is more transparent to them).

dpalomino commented 7 years ago

@wonderchook Are there use cases that wouldn't be covered by a user's ability to export a project then import it on another platform instance? Enabling any private inter-environment routes would breach our isolation of environments, and I'm not sure we should be routing project data over public internet once it's in the platform (if the user chooses to do so via export/import, I think that exposure is more transparent to them).

At least for now I think with the export/import functionality we should be fine for moving projects from demo to platform (it is not a very common scenario). It would be nice though to have a workaround to download the xlsform used in the original project (now when the project already contains some data it's not possible to download it):

screen shot 2017-04-21 at 18 58 01

amplifi commented 7 years ago

@dpalomino The xlsform download sounds like it should be a user-facing option. It'd be good to have a separate issue for tracking that.

dpalomino commented 7 years ago

ok, thanks @amplifi. I will open it now.

clash99 commented 7 years ago

THESE TWO WIREFRAMES ARE OUTDATED - SEE FINAL SET OF WIREFRAMES DATED 07/13/17

Here are two options for bulk archiving of projects by a super user. The first option is more scalable since it also allows for the additional delete option. Initially only the archive action would be available in the dropdown. The second mock only allows for archiving (not truly bulk) but would be quicker development time (I believe). project index - option 1

project index - option 2

wonderchook commented 7 years ago

I'm a little afraid of the 1st one, but I think it is what we need.

What if the archive button was added next to the projects like the 2nd and then the 1st was delete only where you could select multiple? Should we require reauthentication for deleting?

clash99 commented 7 years ago

For the first option, I think there should be a confirmation modal for both archive and delete actions. I was even wondering if a project should have to be archived to be eligible to be deleted. Or does that extra step make sense? The case would be to avoid deleting projects that really should just be archived. The flow would be to select "Archived" from the top display dropdown, check the box, select "Delete" from the bottom action dropdown, and then submit. Thoughts?

wonderchook commented 7 years ago

I was thinking a project has to be archived to be deleted too. I think we want to make it pretty hard to delete things.

dpalomino commented 7 years ago

Hey @oliverroick. We were talking about implementing in next sprint:

Maybe if you have time for this after #1476?

Thanks!

oliverroick commented 7 years ago

Before I can implement this, we should agree on design and approach for these features.

The ability of tagging projects as "un-deletable", so we can skip the ones used by partners or for demos in the programs team

Ability to remove the other projects in demo

We should apply the processes that we agreed upon in Madrid. Which means we should write a specification to outline what these features will look like.

dpalomino commented 7 years ago

We should apply the processes that we agreed upon in Madrid. Which means we should write a specification to outline what these features will look like.

Thanks @oliverroick. Yes, agreed, as there was a proposal ongoing for the UI (@clash99's comment) I missed this one. I'm working on it today.

dpalomino commented 7 years ago

Hey @oliverroick, @amplifi, @clash99

Can you please take a look and confirm if this properly reflects what we've been discussing? When I have your go I'll create the DR. Happy to discuss in a call as well.

Thanks!

Problem Statement

Currently both staging and demo environments have many internal test projects that need to be cleaned, as they are impacting on the maintenance of those environments due to performance, stability and potential migrations needed.

We need a minimum set of super user tools to manage this, to be able to remove projects from either staging or demo.

Context / Use case

When testing new features, or working with partners on early versions of questionnaires, we are generating a lot of projects that have no value after this testing and then should be deleted.

However, there may be other projects that are still used in trainings with partners, when working on intermediate versions of questionnaires, or when testing a recently deployed feature. So we would also need a feature to skip those projects of being deleted.

User Stories

Description

User Story 1 - Tagging un-deletable projects

For this feature, I would suggest the UI proposed by @clash99 (see this wireframe), having checkboxes for each project and a drop-down menu including the "Mark as un-deletable" option, and then a submit button.

User Story 2 - Individually remove projects

Similarly we can use the same UI with the drop-down menu including a "Delete" option, and then a submit button.

For this case we should have a confirmation modal.

User Story 3 - Remove all projects skipping "un-deletable" ones

I would suggest not to have a UI for this, but just periodically execute a script for this.

I think we could have two options here:

  1. Use the project creation date for filtering the projects to be deleted
  2. Use the project timestamp

If it is possible and the effort is similar, I think (2) would be preferable, so we are by default skipping those projects that are "active" (i.e. being edited or with new records)

clash99 commented 7 years ago

@dpalomino - I'm a little concerned about the undeletable option. Couple of questions/comments:

dpalomino commented 7 years ago

Thanks @clash99!

Is the person creating the project for the purpose of a demo a superuser? If not then do they communicate with a superuser to mark it as undeletable? I'm confused by the work flow.

I think this is only going to impact to a few bunch of projects, those used for demoing (the ones in the Programs organization) and the ones currently being used for training/testing purposes with partners (not more than 5-6 at a time I think), and either @SteadyCadence or me will be involved, so it is just communicating to a super user when one of this projects need to be marked as undeletable (my estimation is that this is not going to happen more than once a week or so).

I don't like the idea of adding it to the checkbox/ dropdown combo wireframe. I would suggest adding it to the project details page (similar to the public/private toggle) and then when you see it in the index, those projects have a disabled/ no checkbox so their status can't be changed. (This is assuming these projects are more rare so that setting this option one project at a time isn't too time consuming.)

Sure, I was just suggesting this for reusing the wireframe you created. As this is only affecting to super users the UX is not critical, but if you want to include in the project details I agree that this is a better place if that is not adding more work.

I also hate the word undeletable but I can't think of a better one and I guess it is clear what it means! :)

haha! Agree :-) I'm open to suggestions, maybe the "indestructible" projects? :-P

clash99 commented 7 years ago

@dpalomino - The ideal workflow for making these demo projects stick around would be for the system to know you are Cadasta internal staff and provide you with the option of setting it "official/undeletable" when you create it. I don't think we have that mechanism though. Do we?

dpalomino commented 7 years ago

Yes, that would be ideal of course @clash99. But I guess that would mean to add a new permission to the system "Cadasta Staff" to be something intermediate between the super user and a regular user? Would that be interfering with the tutelary work that is ongoing? Or can we whitelist a set of Cadasta users just for this?

Thoughts @bjohare? Do you think this would interfere in the (get-rid-of) tutelary work?

amplifi commented 7 years ago

A few quick items:

1) We already have a field in the user accounts table for is_staff. We are missing a UI to enable a superuser to mark other user accounts as Cadasta staff.

2) It should be a superuser marking demo projects to be saved/excluded from deletion by request only, not an end user or staff member. It shouldn't be easy to retain projects because we want the demo environment model to be impermanent by default. There should be very few exceptions to this rule. The workflow would be to create a GitHub Issue to request that a project be excluded from deletion, assigned to a superuser.

3) Superusers should be able to view a checkbox per project and per org on the project and org pages, respectively, to enable batch selections with a single save button (like Option 1 wireframe above). If a project is marked for exclusion from deletion, the corresponding org should be marked automatically. Similarly, if an org has only one project, and the project is marked for deletion, the org should be marked automatically.

4) There should be a badge or similar icon visible to all users to indicate on the orgs list, projects list, and project overview page that it has been marked for exclusion.

amplifi commented 7 years ago

@dpalomino Re: user stories:

We can call it "Retain." Retain/Archive/Delete in the drop-down.

There should be a confirmation modal for each action (not just US2), showing the action and the selected items.

For US3, we will not be deleting projects via script, nor will we be running random scripts against live production databases. Deletion of obsolete demo projects should be performed as part of the release process. Eligibility for deletion should be determined based on project creation date, not when it was last updated -- there should be no need to use a demo project more than 30 days without it being requested for retention.

dpalomino commented 7 years ago

Thanks @amplifi

We already have a field in the user accounts table for is_staff. We are missing a UI to enable a superuser to mark other user accounts as Cadasta staff.

Ah, thanks for letting us know.

It should be a superuser marking demo projects to be saved/excluded from deletion by request only, not an end user or staff member. It shouldn't be easy to retain projects because we want the demo environment model to be impermanent by default. There should be very few exceptions to this rule. The workflow would be to create a GitHub Issue to request that a project be excluded from deletion, assigned to a superuser.

That's fine for me. I was suggesting to avoid overloading the superusers, but it's going to be really few projects in this list. I can update the requirements to reflect this.

Superusers should be able to view a checkbox per project and per org on the project and org pages, respectively, to enable batch selections with a single save button (like Option 1 wireframe above).

Ok, but I think it's going to be very few projects, so I think batch selections are not going to be critical. But if we are not going to use the is_staff field, it's true that the batch selection makes more sense here (unless it'd be much more complicated to implement, but I don't think it'd be the case).

If a project is marked for exclusion from deletion, the corresponding org should be marked automatically. Similarly, if an org has only one project, and the project is marked for deletion, the org should be marked automatically.

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

There should be a badge or similar icon visible to all users to indicate on the orgs list, projects list, and project overview page that it has been marked for exclusion.

Good point, thanks! And I'd say also for the organization overview page. I can include this in the requirements (similar badge to the private or archived ones).

screen shot 2017-07-12 at 15 18 21

We can call it "Retain." Retain/Archive/Delete in the drop-down.

Retain, great. That sounds good ;-)

amplifi commented 7 years ago

@dpalomino

Ok, but I think it's going to be very few projects, so I think batch selections are not going to be critical. But if we are not going to use the is_staff field, it's true that the batch selection makes more sense here (unless it'd be much more complicated to implement, but I don't think it'd be the case).

The purpose behind the batch/checklist format isn't just to simplify a large volume of operations. It provides a single, unified interface for managing projects and organizations. It prevents superusers from having to load project data (by loading a project's dashboard page) just to retain/archive/delete a project. It reduces staff time spent on these requests by shrinking the superuser workflow from 1) find a project in the list, click on a project link, load the project data, mark it for retention, confirm the action on the modal, return to the project page, repeat for each project and org... to 2) load the project page, tick the box(es) for the desired project(s), select the action from a drop-down, confirm the modal. It minimizes context-switching for the superuser. The single batch interface for retain/archive/delete is less to QA, has fewer potential points of failure to maintain going forward, and is actually less work to implement.

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

No; this is essential. It's trivial to add these conditional requirements and reduces the risk of data corruption. If someone were to mark a project for retention, but not mark the organization, we could end up in a situation where an org has been deleted but its project has not. (Of course, the deletion functionality should prevent deletion of orgs with existing projects, but we shouldn't only be putting this critical check in one place. We should be requiring data consistency across the platform.)

Good point, thanks! And I'd say also for the organization overview page. I can include this in the requirements (similar badge to the private or archived ones).

Yes, I included the organizations list page in my comment above.

clash99 commented 7 years ago

@dpalomino Might be dumb questions but trying to think through the logic scenarios and want to be sure I understand:

clash99 commented 7 years ago

Wireframes

Project Index

project index

Organization Index

organization index

Warning Messages for creating new org or project

warning messages

These don't address issues for management of the user area (#983). I wasn't sure if that was part of this project but I will add those wireframes in that issue.

amplifi commented 7 years ago

@clash99

If a project is marked as deleted, does it disappear from the list immediately (after confirmation)?

A project should disappear from the list when it has been deleted. Confirming any operation (archive, retain, delete) on this page should refresh the view. When the project is deleted, it will automatically no longer appear in the project list. Similar to how when a project is archived, clicking to confirm the operation refreshes the view and the archived project has the corresponding badge next to it.

Are archived projects able to be marked to retain? If so, does that change their status to active?

Yes, archived projects can be marked for retention. No, it shouldn't change their status to active; they should remain archived. The importance of keeping "Archived" as a distinct state comes into play in the next question, as well.

Does a project or organization have to be archived first to be deleted?

No, this shouldn't be required. "Archived" and "able to be deleted" should remain distinct states, because they could mean very different things across environments. In production, there's no expectation of deletion, so archiving implies its own context.

What happens if a user tries to archive or delete an organization that contains a retained project?

This should never be possible. Item #3 in my comment above would preclude it: If a project is marked for exclusion from deletion, the corresponding org should be marked automatically.

amplifi commented 7 years ago

@clash Wireframes look great (as always)! To address the questions from them:

Re: wireframe pg. 1 no. 7, the dialogue for the retention modal should use a different word than "Active", as archived projects can also be marked to retain.

Re: pg. 3 no. 1, I'd use the number "30" and bold the clause "all projects will be removed after 30 days."

Otherwise, 👍

dpalomino commented 7 years ago

Thanks @amplifi for the feedback!

The purpose behind the batch/checklist format isn't just to simplify a large volume of operations. It provides a single, unified interface for managing projects and organizations. It prevents superusers from having to load project data (by loading a project's dashboard page) just to retain/archive/delete a project. It reduces staff time spent on these requests by shrinking the superuser workflow from 1) find a project in the list, click on a project link, load the project data, mark it for retention, confirm the action on the modal, return to the project page, repeat for each project and org... to 2) load the project page, tick the box(es) for the desired project(s), select the action from a drop-down, confirm the modal. It minimizes context-switching for the superuser. The single batch interface for retain/archive/delete is less to QA, has fewer potential points of failure to maintain going forward, and is actually less work to implement.

Ok, if it's even easier to implement then I think we all agree...

I think this would be a nice to have, but as I mentioned, the list is going to be short, so we can easily provide with the list of projects and organizations to be "retained" in the github issue. IMHO I think we could skip this (unless it'd be something trivial to implement). I can include it in the requirements as an optional nice-to-have thing. Sounds good?

No; this is essential. It's trivial to add these conditional requirements and reduces the risk of data corruption. If someone were to mark a project for retention, but not mark the organization, we could end up in a situation where an org has been deleted but its project has not. (Of course, the deletion functionality should prevent deletion of orgs with existing projects, but we shouldn't only be putting this critical check in one place. We should be requiring data consistency across the platform.)

Ok, if this is not adding much complexity to the implementation I think it'd be useful.

And thanks @clash99! Answering the questions:

If a project is marked as deleted, does it disappear from the list immediately (after confirmation)?

Yes.

Are archived projects able to be marked to retain? If so, does that change their status to active?

Yes.

Does a project or organization have to be archived first to be deleted?

No, I think that's not necessary.

What happens if a user tries to archive or delete an organization that contains a retained project?

It'd not be allowed. The organization cannot be deleted when there are projects within the organization. And the organization should not be archived when there are active projects within the organization.

I think a simple error message for this would be sufficient.

dpalomino commented 7 years ago

Thanks @clash99 for the wireframes! Looks really great :-)

clash99 commented 7 years ago

Thanks @amplifi and @dpalomino for the feedback. The 7/13/17 wireframes above have been updated. Changes include:

Let me know if you see anything else that needs updating.

oliverroick commented 7 years ago

Ok, so this discussion got a bit out of hand. This needs a formal decision record, including a functional and technical spec. Otherwise, development will be a big, untestable mess.

I'll lead on creating the decision record; we're aiming to finish and approve the DR for this sprint, development will have to wait for the next sprint.

dpalomino commented 7 years ago

Hey @oliverroick, @clash99, @amplifi

I've updated the description with the final version of the requirements including all the feedback (thanks!) received. Please let me know if I'm missing something.

I'll lead on creating the decision record; we're aiming to finish and approve the DR for this sprint, development will have to wait for the next sprint.

Yes, that sounds reasonable to me... thanks @oliverroick!