Allow organisations to be closed.

[rossjones]

Although it is possible to delete organisations, there is no functionality to allow for organisations to be closed. Closed organisations are those that cease to exist, but whose datasets and other metadata are still important.

Sometimes the organisation's datasets are merged with another, sometimes they're just needed for a historical archive, but deleting them (and thereby removing from public view) is not what is required.

I'd suggest we'd need:

• Allow for marking an organisation as closed (or re-opening).
• Prominently show closed organisations as closed (optionally with a link to the new organisation it may have become part of).
• Show warning when adding to, or editing, datasets in closed organisations.
• Enable moving of an organisation's datasets to another organisation wholesale. Moved datasets should contain metadata about where they have come from.
• Ensure that whenever the organisation is shown, the title is rendered with _("(Closed)") appended, or some other way of ensuring that the title clearly shows the organisation no longer exists.

[wardi]

:+1: should we have a standard "closed_reason" list too?

• Inactive
• Replaced by X, Y...
• Mirror (harvested from org that lives elsewhere not for updating here)
• Locked by administrator
• ...

[rossjones]

The most useful reason is 'Replaced by' especially if there is a link to the replacement organization, but I think inactive is already implied by the closed.

As far as I am aware, but need to check, ckanext-harvest should already take care of not allowing editing of ckanext-harvest, at least that's what we've implemented on DGU, so I hoping it is default functionality - having the reason won't hurt though and might simplify other code elsewhere.

Locked by administrator isn't clear to me, it doesn't explain why it was closed, so perhaps we can skip that one?

I started a PR using a field on the model, but think I'll probably put the reason and redirect in as extras.

Another thing I wanted to discuss was whether we thought this could be done (as I have in the PR) as a field on the model, or whether it would be better as a collection of states where states == ['active', 'closed', 'replaced', 'deleted'] ? Having a state of replaced would imply an extra called 'replacement-organization' or something similar so that it could be taken into account. One problem with the many states suggestion is that 'closed' and 'replaced' will have to be actively handled in several places where there is a check on 'active'.

[adamamyl]

The most useful reason is 'Replaced by' especially if there is a link to the replacement organization, but I think inactive is already implied by the closed.

Do you think some automatic redirection would be useful here; redirect to new org, and re-house the datasets

[rossjones]

Yes, exactly for that reason, but perhaps not automatic. There's a use-case where we want the organisation closed, still with datasets, but with a bit alert and link to the new organization.

[adamamyl]

Another thing I wanted to discuss was whether we thought this could be done (as I have in the PR) as a field on the model, or whether it would be better as a collection of states where states == ['active', 'closed', 'replaced', 'deleted']

and 'hidden', 'creator_only' if we're looking at states (which I think is a good idea) -- hidden say for if the resource needs to be reviewed

[gjlawran]

I appreciate all the effort on this - but I wonder if business practices wouldn't just be sufficient - rather than code?

If we have an organization that is no longer active - usually the datasets are transferred to other agencies who then manage them as custodians. If the datasets are 'orphaned' by an organization that is no longer active - and not transferred (adopted) - then we would simply eliminate the ability of members to add new packages to that now dormant organization. Typically this would be achieved by removing all the members from the dormant organization.

If we wanted to provide 'redirection' and an explanation for an organization that is now dormant we would just place a note in the description of the organization with a link to the new organization that has superseded it.

[davidread]

@gjlawran I sympathise with this approach - we've been doing this. One key use case that it doesn't cover is that the orphaned datasets are still in the catalogue pages and search results, and the message about the organization being closed is only available on the publisher page. We want to tell people "yes we know the organization is closed now" every time it is mentioned, by calling it 'NHS Bogminster South Health Trust (Closed)' in the search results, and perhaps adding a coloured warning box when you view its dataset.

At the ckan tech meeting today general opinion is:

• yes this should be a feature in ckan core
• yes lets record one (or more?) publisher that 'took over' from the closed one. But let's keep the relationship flexible - maybe allow free text?
• prefer not to mixing the state field with the 'closed' flag - it would disrupt a lot of queries in the code, it might be useful to know an org is closed or not when deleted.
• lets record the date of the change (if available)

[davidread]

When viewing the closed organization, it would be good to have a bold box saying something like: e.g. "This organization was closed in 2012. Responsibility passed to Org A and Org B.".

There are a number of relationships to the follow-on organization. It's probably worth adding the ones we can think of now and making it east to add more. Here's what I can think of:

• simple rename - e.g. "Technology Strategy Board is now called Innovate UK" - it is debatable whether this should be listed - see below
• direct replacement - e.g. "British Waterways" (a government agency) was closed and replaced with a new legal entity "Canal & River Trust" which is a charity, and all of the assets and responsibilities were transferred. I think this is distinct from a simple name change.
• replaced by a class of bodies - e.g. every NHS Primary Care Trust was closed in favour of a new set of Clinical Commissioning Groups covering different areas, and there is no 1:1 mapping of PCTs to CCGs.
• partially replaced - e.g. "NHS Connecting for Health" was abolished and some projects and responsibilities were taken over by the "Health and Social Care Information Centre"
• split up - e.g. "NHS Information Authority" was abolished and its work was divided between the "National Programme for IT" and "Information Centre for Health and Social Care"
• merged into new organization - e.g. "National College for School Leadership" merged with the "Teaching Agency" to become the "National College for Teaching and Leadership". Another example: four bodies were merged to create the "NHS Information Authority".
• merged into / responsibilities passed to an existing organization - e.g. "Postal Services Commission" was merged into (pre-existing) Ofcom.

When a body renames (withouth other major changes) we want a new URL for the body too, so it makes sense to provide a page at the old URL with the redirect. This could be achieved by having 'renamed' in this list, although it doesn't make sense in the db to flag it as "closed". Actually I think a renamed organization or dataset would be better served by a 301 redirect (this is what gov.uk does too) and a list of redirects could be stored in another table.

[gjlawran]

@davidread thanks for your explanations. Could these be restated as:

1) When a request is made for a package that belongs to an organization that is now closed - the state of the organization should be obvious on the package page.

2) When a request is made for the organization list of packages: * http://data.gov.uk/publisher/united-kingdom-hydrographic-office * http://catalogue.data.gov.bc.ca/organization/financial-reporting-and-advisory-services the state of the organization as well as its previous name are presented on the responding organization page.

3) When a request is made for an organization list of packages with the previous name for the organization - the system should do an automatic redirect to the new name - while serving a 301 http response.

IFF I have correctly summarized your requirements: Could 1) be achieved by altering the description fields of packages indicating that they belong to a now closed organization ?

Could 2) be achieved by adding a description indicating the state of the organization and previous name if applicable?

Could 3) be achieved by allowing the system to manage redirects for organizations ? I would expect if the agency was a simple rename we would just change the organization name - but we could setup a redirect if we thought clients were using the old name.

[davidread]

@gjlawran Yes 1 and 2 are key parts of what's needed and the solutions you suggest cover the basic requirements. However I think that by making changes that are planned we can create better visuals for the user, and make it less likely for admins to make mistakes.

Maybe you have easily controlled business processes where you are, but for us the administration of publishers is done by lots of people, and it'd be better to have a 'closed' checkbox rather than a note saying "When an organization closes you need to do X and Y".

[rossjones]

3) When a request is made for an organization list of packages with the previous name for the organization - the system should do an automatic redirect to the new name - while serving a 301 http response.

Auto-redirects may be a useful feature in some cases, but there is a case where the organization is closed, has been replaced by another but the datasets not moved. These are essentially archived and available only for reference. We don't want to redirect in these cases. Also, the function of some organisations may have been replaced by more than one organization.

[gjlawran]

Are we proposing limiting redirection of organization REST requests

• http://data.gov.uk/publisher/environment-agency

or we wanting redirection of parameterized queries too?

• http://data.gov.uk/data/search?publisher=environment-agency

I expect some users will also bookmark these.

Is redirection instantaneous or delayed so that users are made aware of the redirection? is redirection temporary or permanent? I anticipate agency names (URLs) may wish to be reused.

@davidread rest assured we are blessed with 200+ unique users :) @rossjones just confirming - Org A closes, Org B replaces , but packages from A do not migrate to B? So in this case you want to indicate on the Org A page the closure and institution replacement, but the packages continue to be available from the closed Org A , and no-redirection.

[rossjones]

just confirming - Org A closes, Org B replaces , but packages from A do not migrate to B? So in this case you want to indicate on the Org A page the closure and institution replacement, but the packages continue to be available from the closed Org A , and no-redirection.

@gjlawran exactly that, with a link to the new replacement in the message that notifies you that it is no longer active.

[davidread]

• closed flag
• link to another group
• relationship with this other group - free text or drop-drop down (default to something anodyne like "took over from")
• date when closed (optional)