Open morrisonnorman opened 5 years ago
Quick look at what donors are in what bundles...
bundle_uuid | donor_organism.provenance.document_id | donor_organism.is_living |
---|---|---|
9914ac5e-772f-4347-ae33-290bd67ddf11 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
1076c830-8986-4fee-9e7a-847fe0b227c7 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
6d9faabd-5da1-4643-8f02-bce67d9fc9bb | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
0b80acf7-2295-409f-9f33-3f05fc3f4158 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
227f2010-a6df-4c1b-9fb7-fc6cedeff8ce | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
2165e389-4760-4e89-98a8-49dbc0af00d0 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
0cc25d1d-6218-4dad-ba5f-91ac4144a24b | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
26dadf9e-4690-48df-b7f1-eac2b50c1462 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
186c2fc5-fcc4-4de4-b577-179cf73c0d94 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
7e330852-f693-4303-bc83-09e6d145d737 | 26f8b5cf-79a2-441f-a71a-a272a06c4a78 | no |
10ae73ef-d2e1-4a1d-a192-cf332d850c4f | 2a7fbc94-aba3-431d-9156-a1c613e1e681 | no |
71a582dc-6a03-4bb4-a4f7-29bbf289d8b9 | 2a7fbc94-aba3-431d-9156-a1c613e1e681 | no |
67bd4c99-a5f8-4607-8571-9c97fed0832e | 2a7fbc94-aba3-431d-9156-a1c613e1e681 | no |
b44a0842-b2c0-4392-91f7-57e312c0507e | 2a7fbc94-aba3-431d-9156-a1c613e1e681 | no |
4717eb27-350f-41c1-adc2-3589e17048ee | 3c7b6d6f-9502-4851-9668-52a433f4a3b2 | no |
b6037eec-60da-48da-85a4-b7438539a6ae | 3c7b6d6f-9502-4851-9668-52a433f4a3b2 | no |
192ccf23-5847-458d-9527-c730677e7093 | 3c7b6d6f-9502-4851-9668-52a433f4a3b2 | no |
5c4bb0a6-705e-43a1-9714-fc7a60334bed | 3c7b6d6f-9502-4851-9668-52a433f4a3b2 | no |
a44844d6-2728-4595-9fa7-bb6f2a34b629 | 5b56117b-5170-4d7f-8ce7-674147a58a72 | no |
0d6d1122-acf0-466d-ab0b-a131edfb663e | 5b56117b-5170-4d7f-8ce7-674147a58a72 | no |
3013e0db-e471-4134-9e0c-b3bed2464a8e | 5b56117b-5170-4d7f-8ce7-674147a58a72 | no |
368cd6c7-515b-4173-9703-8a95796bba62 | 5b56117b-5170-4d7f-8ce7-674147a58a72 | no |
f7081630-0d97-4550-8b16-77008ea868a3 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
2e373a6e-f531-449b-a989-eb163322ce83 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
a36a76dd-3036-42e6-beb2-dd41e1e312b6 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
b7e88f9e-45e4-4472-aa77-fc46886614a4 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
8cd7c4db-675b-4412-bb60-df0340245828 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
6a6bd6e3-af01-4e0c-81ad-97eb0bc15db5 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
c02ec92e-1aa0-49ae-a2fd-ceeb4b0cbdb3 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
08ca411d-c58a-487a-b7da-4d5e83732252 | 87316bb3-ad5e-4921-8f5f-29acad300ad2 | no |
5a37f51b-3f66-4b22-95c1-d82842265f19 | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
4accc284-186c-4b81-9810-29314a4a6948 | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
9eb1a782-4c15-42b7-8d4a-e6dfca346fe3 | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
6529d0e9-22ff-4276-99ca-884e45a5ceb3 | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
2e478e45-e8e5-402a-b41f-5a9c2f2edd3a | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
89a909ff-43e2-4ef9-832a-11ea6ab7689a | 98961f76-94cd-4848-8710-d2ce8edab34f | no |
e22e7f03-4917-4ab9-ba0c-31784fa62de9 | d209d8fc-fd2c-452c-82e3-a81652276884 | no |
5c4b214d-ffd7-426b-88a2-cf611e5803f7 | d209d8fc-fd2c-452c-82e3-a81652276884 | no |
4420213f-8c68-4559-8d76-b583820a1ff3 | d209d8fc-fd2c-452c-82e3-a81652276884 | no |
284679d1-6b69-4d52-8774-c13943610299 | d209d8fc-fd2c-452c-82e3-a81652276884 | no |
ad88924f-248f-4c6e-bad9-d64495b03de1 | d3387a14-d06d-4b2b-bc9b-e7ece3be964a | no |
3b17e3c9-5f1c-49c6-b4fc-e3d658e7b731 | d3387a14-d06d-4b2b-bc9b-e7ece3be964a | no |
6ff1a6f0-f214-45bc-878a-f8919cb3e581 | d3387a14-d06d-4b2b-bc9b-e7ece3be964a | no |
3bab143b-606a-4890-afa1-7dfa92370cf7 | d3387a14-d06d-4b2b-bc9b-e7ece3be964a | no |
499b690e-feee-4675-8531-882a62b6020c | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
fff16441-5212-4319-bf52-9e23d0eff7eb | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
5b8fc043-b00e-4699-b75f-19be0c7d9dc0 | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
ce1f6515-6143-4e77-b162-fe6060cc10e6 | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
4882ec6f-fe81-43d8-9446-595fbf6319da | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
27a43436-6e63-44ce-a2b3-80618f00b7f6 | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
0d7aee1d-a3d7-4532-903a-c531ac07998d | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
9ca7449a-4b91-4cdd-b226-a7fbb7b1ffd0 | 191eed41-c398-4911-b2d8-63f110a0e823 | yes |
8b7635a4-88a4-4c76-9946-6e3bebd51bfa | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
1a26883e-d034-491c-8d0a-0e93e854eacb | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
daaf63cf-04ae-4335-868c-32e63af0c37a | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
6f01e3ae-9743-4424-983b-18b18f0ffeed | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
002fe390-ba3f-4aa7-ae43-aa7949b09bb1 | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
a1e2d3d2-3b47-4d20-a331-548818f3604b | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
caf3a1b6-60d3-42fd-a56c-48fe0e6de571 | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
79ec8f41-fccc-46ed-af43-e63892238d77 | 1e851cf7-106f-45dc-b65f-222c805bc961 | yes |
a0a78a4e-7891-4ef2-a22d-1bfab43ffe47 | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
6086e0e1-c679-44ad-bc62-57875b216fc4 | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
7b4fd692-c38d-43c6-bb5b-bf3dcbc837eb | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
f269045b-1b3b-40d6-86ae-398a5b699e1d | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
c0e82caf-6fed-4524-9267-d97ac0a4d2f2 | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
f6d920a5-63ab-4d4b-8fb7-0fe2fdbf2fcd | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
67cdc6a8-9f47-4dd9-908d-9654e489361d | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
97735720-4c1f-4b69-9463-e6c9febc2764 | 52e1680e-4a73-46ed-ba7c-8e52de2aa04e | yes |
f838b9fe-3e03-41cf-8e1f-1f17db456975 | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
3acc603d-b617-49ae-aa0d-353db0aa9e7e | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
880a3226-d432-4ab7-b81a-4c00421c6590 | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
32b6880e-6c5c-41e4-b5b1-96b097cc52da | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
c99ea041-0daf-4b25-bee9-d9691af11a5e | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
5bf86ddb-a57b-49be-b734-8b09d7259e60 | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
7ef3700c-d2d3-4e6e-b73e-c22f3380e7c3 | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
70372f87-4cd4-40d7-8c09-1ebc641e1384 | 5e31ffe8-a7b2-4478-adc1-4f7368339e71 | yes |
d8f7f78d-1c77-49cc-b934-28d0da11c6d0 | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
8388fed2-1af2-4102-a578-82d3ef5ace03 | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
9d993f96-67d2-4640-8217-89b1ecc60028 | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
6963e95a-f81d-4bfc-9639-46da2a958d7a | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
f840f06a-58b1-4699-a672-ef0b08fb3304 | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
8f585ec2-54c2-4d7b-8cea-0a9ceada6c8f | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
88aed2c9-6537-40fe-a04c-cd58900e9abf | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
a182995c-83c9-4da9-8029-f73b1d742d1b | 6ce8c2a2-d013-4bdd-8444-3bba7994125a | yes |
00005cce-dfbe-40a9-922a-e1da11a11ada | a9efb8f5-fd92-4f49-9fc8-18310979752b | yes |
ea25386a-dc17-453d-9312-b61995f2c3f6 | a9efb8f5-fd92-4f49-9fc8-18310979752b | yes |
328a2bf0-3f3c-4441-96a8-ddf41852e650 | a9efb8f5-fd92-4f49-9fc8-18310979752b | yes |
363aeb9a-010d-49ec-b8e0-9062552b9062 | a9efb8f5-fd92-4f49-9fc8-18310979752b | yes |
3c3061a7-1dc5-422e-bc14-4760b47f6ad2 | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
81c5a0ed-50d3-4c60-b1b4-57df9862b636 | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
62893d1d-91ea-4842-8d76-d6d4e67a795e | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
0dce2a49-1d81-45a2-98c3-75c1e154e96e | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
9e91edba-447e-47d3-8095-4406520e2fe9 | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
9bd51e8f-abb2-40e6-8ee2-e0afa8b6caa6 | b4997866-8e80-49af-823a-83d24aaa2d83 | yes |
@morrisonnorman are all 7 living donors European?
@ESapenaVentura Can you confirm for @jahilton which of the 7 living donors are European?
Previous quote from Enrique: "It seems that donors txk2, txk3, pRCC, RCC1, RCC2, RCC3 and VHL are alive".
@jahilton All of the living donors are European.
Here's our living donor bundle list
499b690e-feee-4675-8531-882a62b6020c
fff16441-5212-4319-bf52-9e23d0eff7eb
5b8fc043-b00e-4699-b75f-19be0c7d9dc0
ce1f6515-6143-4e77-b162-fe6060cc10e6
4882ec6f-fe81-43d8-9446-595fbf6319da
27a43436-6e63-44ce-a2b3-80618f00b7f6
0d7aee1d-a3d7-4532-903a-c531ac07998d
9ca7449a-4b91-4cdd-b226-a7fbb7b1ffd0
8b7635a4-88a4-4c76-9946-6e3bebd51bfa
1a26883e-d034-491c-8d0a-0e93e854eacb
daaf63cf-04ae-4335-868c-32e63af0c37a
6f01e3ae-9743-4424-983b-18b18f0ffeed
002fe390-ba3f-4aa7-ae43-aa7949b09bb1
a1e2d3d2-3b47-4d20-a331-548818f3604b
caf3a1b6-60d3-42fd-a56c-48fe0e6de571
79ec8f41-fccc-46ed-af43-e63892238d77
a0a78a4e-7891-4ef2-a22d-1bfab43ffe47
6086e0e1-c679-44ad-bc62-57875b216fc4
7b4fd692-c38d-43c6-bb5b-bf3dcbc837eb
f269045b-1b3b-40d6-86ae-398a5b699e1d
c0e82caf-6fed-4524-9267-d97ac0a4d2f2
f6d920a5-63ab-4d4b-8fb7-0fe2fdbf2fcd
67cdc6a8-9f47-4dd9-908d-9654e489361d
97735720-4c1f-4b69-9463-e6c9febc2764
f838b9fe-3e03-41cf-8e1f-1f17db456975
3acc603d-b617-49ae-aa0d-353db0aa9e7e
880a3226-d432-4ab7-b81a-4c00421c6590
32b6880e-6c5c-41e4-b5b1-96b097cc52da
c99ea041-0daf-4b25-bee9-d9691af11a5e
5bf86ddb-a57b-49be-b734-8b09d7259e60
7ef3700c-d2d3-4e6e-b73e-c22f3380e7c3
70372f87-4cd4-40d7-8c09-1ebc641e1384
d8f7f78d-1c77-49cc-b934-28d0da11c6d0
8388fed2-1af2-4102-a578-82d3ef5ace03
9d993f96-67d2-4640-8217-89b1ecc60028
6963e95a-f81d-4bfc-9639-46da2a958d7a
f840f06a-58b1-4699-a672-ef0b08fb3304
8f585ec2-54c2-4d7b-8cea-0a9ceada6c8f
88aed2c9-6537-40fe-a04c-cd58900e9abf
a182995c-83c9-4da9-8029-f73b1d742d1b
00005cce-dfbe-40a9-922a-e1da11a11ada
ea25386a-dc17-453d-9312-b61995f2c3f6
328a2bf0-3f3c-4441-96a8-ddf41852e650
363aeb9a-010d-49ec-b8e0-9062552b9062
3c3061a7-1dc5-422e-bc14-4760b47f6ad2
81c5a0ed-50d3-4c60-b1b4-57df9862b636
62893d1d-91ea-4842-8d76-d6d4e67a795e
0dce2a49-1d81-45a2-98c3-75c1e154e96e
9e91edba-447e-47d3-8095-4406520e2fe9
9bd51e8f-abb2-40e6-8ee2-e0afa8b6caa6
We don't have explicit data about the donor ethnicity, although it's safe to assume that all of them are European. I can look at the paper for further confirmation if necessary
@ESapenaVentura if there's any uncertainty, can you contact Benjamin?
RE European donors: The samples were all collected from British Hospitals, so they are all residents in Europe and subject to GDPR. All data associated data needs to be removed.
@jahilton Although he priority is to delete the 'living bundles', we don't have a mechanism to delete specific bundles and update the metadata for the project.
I think it will have to be a 'delete all' from the project... and reingest.
Worth noting that European in the GDPR context isn't about ethnicity but about residency. I think we have to assume that any biopsy collected by a European institution is covered by GDPR
Instead of reingesting, it should be possible to write a one off program to edit the bundles and create new version, then delete the old versions.
Norman Morrison notifications@github.com writes:
RE European donors: The samples were all collected from British Hospitals, so they are all residents in Europe and subject to GDPR. All data associated data needs to be removed.
@jahilton Although he priority is to delete the 'living bundles', we don't have a mechanism to delete specific bundles and update the metadata for the project.
I think it will have to be a 'delete all' from the project... and reingest.
-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/HumanCellAtlas/dcp/issues/536#issuecomment-543204937 RE European donors: The samples were all collected from British Hospitals, so they are all residents in Europe and subject to GDPR. All data associated data needs to be removed.
@jahilton Although he priority is to delete the 'living bundles', we don't have a mechanism to delete specific bundles and update the metadata for the project.
I think it will have to be a 'delete all' from the project... and reingest.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.*
@diekhans It would be great if there's a better way to address the issue that doesn't involve reingestion in a timely manner
The context in this case is to delete bundles that contain the aforementioned donors, but the rest of the bundles (in case we can AUDR and not re-ingest) will be unaffected by this. Even if they remain unaffected, would you need new versions of them? @diekhans
@ESapenaVentura do the project records specifically reference the cancer biopsys or link to the donors which will be deleted in a manner that isn't fixable with simple updates?
I don't think I understand the question. Do you mean if the project metadata references the cancer biopsies?
Yes, will the project level metadata be made inaccurate or any links in the deceased donor bundles be made invalid when the cancer biopsy bundles are deleted.
Those would be potential things which might require re-ingestion rather than deleting the bundles which reference the alive donors
If no deceased donor bundles reference living donors and any project-level metadata that references the cancer biopsy donors can be updated via simple updates then we might not need re-ingestion
Okay based on discussions in slack and in person with Tony and Alegria there are other reasons that just deleting the needed bundles might lead to challenges later so I suspect this isn't a suitable solution
Tickets have been created & linked to this Epic for Ingest DataStore Browser MatrixService SecondaryAnalysis
@parthshahva does not believe Upload maintains any record so action may not be required. He will comment here once confirmed.
Per discussion in #data-wrangling @ESapenaVentura will be cleaning up the corresponding staging area in the wrangler workspace. I've confirmed that the production staging area was cleaned out after the submission/exporting completed and data was copied over to DSS.
Wrangler issue ticket https://github.com/HumanCellAtlas/hca-data-wrangling/issues/344
Wrangler issue ticket https://github.com/HumanCellAtlas/hca-data-wrangling/issues/344
More comprehensive Project Deletion SOP.
Issue
It has been brought to our attention that the [Spatio-temporal immune zonation of the human kidney] (https://data.humancellatlas.org/explore/projects/abe1a013-af7a-45ed-8c26-f3793c24a1f4) project contains data from living European donors. ALL instances of data associated with living European donors needs to be removed/deleted from DCP components.
High Priority Task List
Remove all instances of data associated with living European donors from DCP components. Deleted from:
A more detailed list of tasks and nascent Project Deletion SOP is being developed in this google doc
11/15/19 More comprehensive Project Deletion SOP.
Requirements
The old project UUID should be maintained as this is already being referenced by 3rd parties. The project can be reingested once the living European data has been removed. The reingested data needs to be linked* to the the old project UUID.
AUDR
AUDR capability does not currently handle partial deletion of data from projects. Project deletion involves removing the entire project dataset and associated bundles. Reingest and manually assign the existing project UUID to the newly ingested project.
Next steps
Once the high priority deletion task list has been completed, the project can be reingested - with the living European donor metadata and data removed. Note that the reingested data needs to be linked* to the the old project UUID.
*MVP for reingestion is a direct replacement of the project UUID with no versioning.