neuropoly / data-management

Repo that deals with datalad aspects for internal use
4 stars 0 forks source link

Migrate this repo's content and shut it down #141

Open kousu opened 2 years ago

kousu commented 2 years ago

The .md files in this repo should migrate to https://github.com/neuropoly/neuropoly-internal-docs/. The scripts/* should be attached to their under code/* in each dataset (and ideally we should make a habit of making sure our curation scripts are written to handle being run from code/). The wiki should be split up -- most of its content is better treated as Issues most of which we've closed by working with datalad/git-annex for about a year now.

The Issues can stay for the moment because it's good to have a forum to discuss these issues in public.

The pseudo-Issues (#135, #108, #116, #85) that are actually PRs for the internal datasets should eventually be handled by gitolite pr (https://github.com/neuropoly/data-management/issues/86). I guess they can stay until we have a better solution for them.

joshuacwnewton commented 2 years ago

The Issues can stay for the moment because it's good to have a forum to discuss these issues in public.

The pseudo-Issues (#135, #108, #116, #85) that are actually PRs for the internal datasets should eventually be handled by gitolite pr (#86). I guess they can stay until we have a better solution for them.

I've spent a couple of minutes adding some tags that might help filter this repo a little better, but... I'm not sure how useful this actually is now that I've done it. :sweat_smile:

kousu commented 2 years ago

Now that https://github.com/neuropoly/intranet.neuro.polymtl.ca/pull/21 is live, I feel I can do this.

First step: use https://github.com/newren/git-filter-repo to extract the documentation from this repo, and put it into the matching place in that repo:

p115628@joplin:~$ git clone https://github.com/neuropoly/data-management/
Cloning into 'data-management'...
remote: Enumerating objects: 509, done.
remote: Counting objects: 100% (408/408), done.
remote: Compressing objects: 100% (300/300), done.
remote: Total 509 (delta 232), reused 162 (delta 93), pack-reused 101
Receiving objects: 100% (509/509), 295.95 KiB | 1.44 MiB/s, done.
Resolving deltas: 100% (262/262), done.
p115628@joplin:~$ cd data-management
p115628@joplin:~/data-management$ pip3 install git-filter-repo
Collecting git-filter-repo
  Downloading git_filter_repo-2.34.0-py2.py3-none-any.whl (99 kB)
     |████████████████████████████████| 99 kB 13.2 MB/s 
Installing collected packages: git-filter-repo
Successfully installed git-filter-repo-2.34.0
p115628@joplin:~/data-management$ ls -l
total 36
-rw-r--r-- 1 p115628 domain users 11038 Nov 17 22:26 git-annex.md
-rw-r--r-- 1 p115628 domain users 15770 Nov 17 22:26 internal-server.md
-rw-r--r-- 1 p115628 domain users   338 Nov 17 22:26 README.md
drwxr-xr-x 2 p115628 domain users  4096 Nov 17 22:26 scripts
p115628@joplin:~/data-management$ git filter-repo --path-match git-annex.md --path-match internal-server.md --to-subdirectory-filter computing-resources/neuropoly/data/
Parsed 144 commits
New history written in 0.12 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at ff0a462 Document what a "release" is and how to work them (#130)
Enumerating objects: 182, done.
Counting objects: 100% (182/182), done.
Delta compression using up to 128 threads
Compressing objects: 100% (34/34), done.
Writing objects: 100% (182/182), done.
Total 182 (delta 86), reused 182 (delta 86)
Completely finished after 0.21 seconds.
p115628@joplin:~/data-management$ ls -l
total 4
drwxr-xr-x 3 p115628 domain users 4096 Nov 17 22:27 computing-resources
p115628@joplin:~/data-management$ tree
.
└── computing-resources
    └── neuropoly
        └── data
            ├── git-annex.md
            └── internal-server.md

3 directories, 2 files
p115628@joplin:~/data-management$ git log -p
commit ff0a462974a21ecce16b95a0e2f43f987411173a (HEAD -> master)
Author: Nick <nick.guenther@polymtl.ca>
Date:   Tue Sep 28 16:06:57 2021 -0400

    Document what a "release" is and how to work them (#130)

    Co-authored-by: Julien Cohen-Adad <jcohen@polymtl.ca>

diff --git a/computing-resources/neuropoly/data/internal-server.md b/computing-resources/neuropoly/data/internal-server.md
index 8a4f942..ee5dacc 100644
--- a/computing-resources/neuropoly/data/internal-server.md
+++ b/computing-resources/neuropoly/data/internal-server.md
@@ -19,8 +19,9 @@ Table of Contents
   * [Upload](#upload)
   * [Reviewing Pull Requests](#reviewing-pull-requests)
     * [Commit Rights](#commit-rights)
-    * [Committing](#committing) 
+    * [Committing](#committing)
   * [New repository](#new-repository)
+  * [Releases](#releases)
   * [Permissions](#permissions)
   * [Renaming](#renaming)
   * [Deletion](#deletion)
@@ -262,6 +263,40 @@ $ git remote add origin git@data.neuro.polymtl.ca:zamboni/project1
 $ git push origin

....


Second step: get it over to where I'm working (my laptop's git was too old to run git-filter-repo, so I did that part on `joplin`)...

kousu@ail:~/src/neuropoly/website$ git clone p115628@joplin:~/data-management Cloning into 'data-management'... remote: Enumerating objects: 182, done. remote: Counting objects: 100% (182/182), done. remote: Compressing objects: 100% (34/34), done. remote: Total 182 (delta 86), reused 182 (delta 86) Receiving objects: 100% (182/182), 32.92 KiB | 3.66 MiB/s, done. Resolving deltas: 100% (86/86), done. kousu@ail:~/src/neuropoly/website$ cd data-management/ kousu@ail:~/src/neuropoly/website/data-management$ ls -l total 4 drwxrwxr-x 3 kousu kousu 4096 Nov 17 22:31 computing-resources kousu@ail:~/src/neuropoly/website/data-management$ tree . └── computing-resources └── neuropoly └── data ├── git-annex.md └── internal-server.md

3 directories, 2 files


Third step: `--allow-unrelated-histories`

(.venv) kousu@ail:~/src/neuropoly/website/neuropoly-internal-docs/computing-resources/neuropoly$ git remote add git-annex-docs ~/src/neuropoly/website/data-management/ (.venv) kousu@ail:~/src/neuropoly/website/neuropoly-internal-docs/computing-resources/neuropoly$ git fetch git-annex-docs warning: no common commits remote: Counting objects: 170, done. remote: Compressing objects: 100% (33/33), done. remote: Total 170 (delta 80), reused 169 (delta 79) Receiving objects: 100% (170/170), 31.97 KiB | 1.52 MiB/s, done. Resolving deltas: 100% (80/80), done. From /home/kousu/src/neuropoly/website/data-management

aaand done.

kousu commented 2 years ago

Oh wow I didn't think a different repo's Fixes: could close something over here. This isn't quite done yet.