search

AllenInstitute / MIES

Multichannel Igor Electrophysiology Suite
https://alleninstitute.github.io/MIES/user.html
Other
22 stars 6 forks source link

Move proprietary repository with MIES here #1

Closed t-b closed 5 years ago

t-b commented 6 years ago
t-b commented 5 years ago

Conversion script:

$cat run.sh 
#!/bin/bash

rm -f /tmp/git
touch /tmp/git

# generate a list of files we want to remove
git log --numstat --name-only --format="" --all --ignore-submodules | grep -v -P ".(ipf|ipfT[0-9]|ibw|ihf|awk|Doxyfile|md|rst|gitignore|hgflow|txt|pxp|sh|nsi|dot|svg|itx|nwb|gitmodules|gitattributes|bat|lnk|py|template|cnf|mailmap|ift|vbs|placeholder|pl|otf|tex|sty|css|clang-format)$" | grep -v -P "(LICENSE|Dockerfile|mies.ico|tools/nircmd/.*|libzmq-.*.dll|ZeroMQ.*\.xop|Arduino-libraries-and-sequencer.zip|MultiClamp700xCommander.*\.xop|itcXOP2.*\.xop|ITCMM.*\.dll|MIESUtils.*\.xop|vc_redist.*\.exe|Packages/ZeroMQ.*)$" | sed -e 's;^Packages/ITC.*;Packages/ITC;' | sed -e 's;^tools/installer/nsis.*;tools/installer/nsis;' | grep -v "^XOP-tango" | sort | uniq > /tmp/files-to-remove.txt

# filter are executed in the predefined and here given order:
#
# index-filter  takes care of removing the unwanted files
# msg-filter    adds the Original-CommitID and fills /tmp/git for the old sha->new sha translation with old sha
# commit-filter fills /tmp/git for the old sha->new sha translation with new sha
git filter-branch                                                                                \
  --index-filter  'git rm --quiet -r --cached --ignore-unmatch $(cat /tmp/files-to-remove.txt)'  \
  --msg-filter    'sed -f /tmp/git;                                                              \
                   echo "\nOriginal-CommitID: ${GIT_COMMIT}";'                                   \
  --commit-filter 'OLD_COMMIT_SHORT=$(echo ${GIT_COMMIT} | cut -c 1-8);                          \
                   echo -n "s/\(${OLD_COMMIT_SHORT}\|${GIT_COMMIT}|${GIT_COMMIT}\)/" >>/tmp/git; \
                   NEW=`git_commit_non_empty_tree "$@"`;                                         \
                   NEW_COMMIT_SHORT=$(echo ${NEW} | cut -c 1-8);                                 \
                   echo "${NEW_COMMIT_SHORT}/g" >> /tmp/git; echo ${NEW}'                        \
  --tag-name-filter cat                                                                          \
  -- --all

# do the old sha->new sha transition on our code base
# could be done as tree filter as well but would make everything *much* slower
sed -i -f /tmp/git Packages/MIES/*.ipf Packages/Testing-MIES/*.ipf

rm -f /tmp/msg

echo -e "Public-Repository: Replace old SHAs" >> /tmp/msg

git commit --allow-empty -F /tmp/msg Packages/MIES/*.ipf Packages/Testing-MIES

rm -f /tmp/msg

# add this script as commit message
echo -e "Public-Repository: Conversion was done with the following script\n\n" >> /tmp/msg
cat ${0} >> /tmp/msg

echo -e "\n\nand the files to be removed where\n" >> /tmp/msg
cat /tmp/files-to-remove.txt >> /tmp/msg

git commit --allow-empty -F /tmp/msg
t-b commented 5 years ago

We can not push the cleaned up repo as it is too large

$git push --force origin master 
Objekte aufzählen: 31109, Fertig.
Zähle Objekte: 100% (31109/31109), Fertig.
Delta-Kompression verwendet bis zu 12 Threads.
Komprimiere Objekte: 100% (7798/7798), Fertig.
Schreibe Objekte: 100% (31109/31109), 111.93 MiB | 2.85 MiB/s, Fertig.
Gesamt 31109 (Delta 24343), Wiederverwendet 29814 (Delta 23253)
remote: Resolving deltas: 100% (24343/24343), done.
remote: warning: File tools/unit-testing/RunAllTests.pxp is 71.43 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File tools/unit-testing/RunAllTests.pxp is 99.04 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File tools/unit-testing/RunAllTests.pxp is 65.35 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File tools/unit-testing/RunAllTests.pxp is 56.29 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File tools/unit-testing/HardwareTests.pxp is 88.86 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File tools/unit-testing/HardwareTests.pxp is 85.59 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 672f7bb55fee4e398c7fd2059473ac6c
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File tools/unit-testing/RunAllTests.pxp is 117.58 MB; this exceeds GitHub's file size limit of 100.00 MB
To github.com:AllenInstitute/MIES.git
 ! [remote rejected]   master -> master (pre-receive hook declined)
error: Fehler beim Versenden einiger Referenzen nach 'git@github.com:AllenInstitute/MIES.git'