search

OpenGATE / GateContrib

User-oriented public repository of Gate (macros, examples and user contributions)
74 stars 135 forks source link

LFS data #50

Closed tbaudier closed 3 years ago

tbaudier commented 3 years ago

Follow PR https://github.com/OpenGATE/GateContrib/pull/48 (@dsarrut @SimonRit )

I open an issue to discuss about LFS data. I gave up the PR to focus on rebase instead of revert.

I push the rebase of @jmletang 's commit but still some data are not correctly followed. Find the list bellow. It's .png files and Materials.db files. For png files I will convert them to LFS. Does for Materials.db need to be LFS too because they are text files?

As we already rebase once I will find all blamed commit and rebase them to have a clean repository.

tbaudier commented 3 years ago

The problematic files:

Encountered 94 file(s) that should have been pointers, but weren't: GATE-RTion/DoseActor/data/HFMaterials2014.db GATE-RTion/DoseActor2-CT/data/HFMaterials2014_test.db GATE-RTion/TPSPencilBeamSource/data/GateMaterials.db GATE-RTion/TPSPencilBeamSource/data/MyMaterials2010.db GATE-RTion/TPSPencilBeamSource/data/MyMaterials2013.db GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db dosimetry/DNA/GateMaterials.db dosimetry/LET/data/GateMaterials.db dosimetry/Radiotherapy/example1/data/GateMaterials.db dosimetry/Radiotherapy/example10/data/GateMaterials.db dosimetry/Radiotherapy/example10/data/phantom-HUmaterials.db dosimetry/Radiotherapy/example11/data/Materials.db dosimetry/Radiotherapy/example12/data/GateMaterials.db dosimetry/Radiotherapy/example12/data/LinacMaterials.db dosimetry/Radiotherapy/example2/data/GateMaterials.db dosimetry/Radiotherapy/example3/data/GateMaterials.db dosimetry/Radiotherapy/example3/data/PreciseMaterials.db dosimetry/Radiotherapy/example3/data/patient-HUmaterials.db dosimetry/Radiotherapy/example4/data/GateMaterials.db dosimetry/Radiotherapy/example5/data/GateMaterials.db dosimetry/Radiotherapy/example6/data/GateMaterials.db dosimetry/Radiotherapy/example6/data/patient-HUmaterials.db dosimetry/Radiotherapy/example7/data/GateMaterials.db dosimetry/Radiotherapy/example8/data/GateMaterials.db dosimetry/Radiotherapy/example9/data/GateMaterials.db dosimetry/TEPCActor/data/myGateMaterials.db dosimetry/UserSpectrum/data/GateMaterials.db dosimetry/doseByRegions/data/AF_GateMaterials.db dosimetry/doseactor/unvoxelized-example/data/GateMaterials.db dosimetry/doseactor/voxelized-phantom/data/GateMaterials.db dosimetry/dosimetry/brachytherapy/data/GateMaterials.db dosimetry/dosimetry/electron_radiotherapy/data/GateMaterials.db dosimetry/dosimetry/external-beam-therapy-photon/data/GateMaterials.db dosimetry/dosimetry/molecular-therapy-I131/data/GateMaterials.db dosimetry/dosimetry/protontherapy/data/GateMaterials.db dosimetry/gaga-phsp/data/GateMaterials.db imaging/CT/Fresnel_FFD/data/GateMaterials.db imaging/CT/XRayRefraction/data/GateMaterials.db imaging/CT/fixedForcedDetectionCT/data/ICRP_HUmaterials.db imaging/ComptonCamera/data/GateMaterials.db imaging/DICOM/mhd-dcm-comparison/data/GateMaterials.db imaging/ExtendedVSource/CommonMacros/GateMaterials.db imaging/ExtendedVSource/Example_1/histograms/EnergyDepositionAnnihilation_pPs.png imaging/ExtendedVSource/Example_1/histograms/EnergyDepositionPrompt_pPs.png imaging/ExtendedVSource/Example_1/histograms/EnergyDeposition_pPs.png imaging/ExtendedVSource/Example_1/histograms/Lifetime_pPs.png imaging/ExtendedVSource/Example_1/histograms/SourcePosition_pPs.png imaging/ExtendedVSource/Example_1/histograms/energy_deposition_spectra_pPs.png imaging/ExtendedVSource/Example_2/histograms/AngleDistributionScattered_oPs.png imaging/ExtendedVSource/Example_2/histograms/AngleDistribution_oPs.png imaging/ExtendedVSource/Example_2/histograms/EnergyDepositionAnnihilation_oPs.png imaging/ExtendedVSource/Example_2/histograms/EnergyDepositionPrompt_oPs.png imaging/ExtendedVSource/Example_2/histograms/EnergyDeposition_oPs.png imaging/ExtendedVSource/Example_2/histograms/Lifetime_oPs.png imaging/ExtendedVSource/Example_2/histograms/SourcePosition_oPs.png imaging/ExtendedVSource/Example_2/histograms/energy_deposition_spectra_oPs.png imaging/ExtendedVSource/Example_3/histograms/EnergyDepositionAnnihilation_Ps.png imaging/ExtendedVSource/Example_3/histograms/EnergyDepositionPrompt_Ps.png imaging/ExtendedVSource/Example_3/histograms/EnergyDeposition_Ps.png imaging/ExtendedVSource/Example_3/histograms/Lifetime_Ps.png imaging/ExtendedVSource/Example_3/histograms/SourcePosition_Ps.png imaging/ExtendedVSource/Example_3/histograms/energy_deposition_spectra_Ps.png imaging/ExtendedVSource/Example_4/histograms/EnergyDepositionAnnihilation_pPs.png imaging/ExtendedVSource/Example_4/histograms/EnergyDepositionPrompt_pPs.png imaging/ExtendedVSource/Example_4/histograms/EnergyDeposition_pPs.png imaging/ExtendedVSource/Example_4/histograms/Lifetime_pPs.png imaging/ExtendedVSource/Example_4/histograms/SourcePosition_pPs.png imaging/ExtendedVSource/Example_4/histograms/energy_deposition_spectra_pPs.png imaging/LUTDavisModel/GateMaterials.db imaging/Optical/GateMaterials.db imaging/ProtonRadiography/data/GateMaterials.db imaging/SPECT_FFD/data/GateMaterials.db imaging/SPECT_GPU/GateMaterials.db imaging/Thermal/GateMaterials.db imaging/UserFluenceSource/data/GateMaterials.db imaging/fastY90/data/GateMaterials.db imaging/fluorescence/data/myGateMaterials.db imaging/garf/data/GateMaterials.db imaging/garf/data/phantom_HUmaterials.db imaging/vpgtle/stage0/data/GateMaterials.db imaging/vpgtle/stage0/data/GateMaterialsElements.db imaging/vpgtle/stage1/data/GateMaterials.db imaging/vpgtle/stage1/data/patient-HUmaterials.db imaging/vpgtle/stage2/data/GateMaterials.db misc/GateMaterials.db misc/GateMaterialsGPU.db misc/TetrahedralMeshGeometry/data/GateMaterials.db misc/TrackLengthFluenceExamples/TLFluence-Example1/Data/GateMaterials.db misc/TrackLengthFluenceExamples/TLFluenceDistribution-Exampl1/Data/GateMaterials.db misc/TrackLengthFluenceExamples/Track_Length_Actors.pdf misc/geometry_STL/Suzanne/GateMaterials.db misc/geometry_STL/kidneys/data/GateMaterials.db misc/hybrid_volume_navigator/MergedVolume/GateMaterials.db misc/hybrid_volume_navigator/Reference/GateMaterials.db

dsarrut commented 3 years ago

I think Materials.db should not be LFS because they are txt files.

tbaudier commented 3 years ago

They are listed in .gitattributes: https://github.com/OpenGATE/GateContrib/blob/master/.gitattributes#L15

... .TABLE filter=lfs diff=lfs merge=lfs -text .db filter=lfs diff=lfs merge=lfs -text

dsarrut commented 3 years ago

maybe there are other .db files that are binaries ?

tbaudier commented 3 years ago

They are all text files:

 ~/Software/GateContrib> file `find . -iname "*.db"`
./GATE-RTion/DoseActor/data/HFMaterials2014.db:                                        ASCII text
./GATE-RTion/DoseActor2-CT/data/HFMaterials2014_test.db:                               ASCII text
./GATE-RTion/TPSPencilBeamSource/data/GateMaterials.db:                                ASCII text
./GATE-RTion/TPSPencilBeamSource/data/MyMaterials2010.db:                              ASCII text
./GATE-RTion/TPSPencilBeamSource/data/MyMaterials2013.db:                              ASCII text
./GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db:                          ASCII text
./dosimetry/DNA/GateMaterials.db:                                                      ASCII text
./dosimetry/EMFields/data/HFMaterials2014.db:                                          ASCII text
./dosimetry/LET/data/GateMaterials.db:                                                 ASCII text
./dosimetry/Radiotherapy/example1/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example10/data/GateMaterials.db:                              ASCII text
./dosimetry/Radiotherapy/example10/data/phantom-HUmaterials.db:                        ASCII text
./dosimetry/Radiotherapy/example11/data/Materials.db:                                  ASCII text
./dosimetry/Radiotherapy/example12/data/GateMaterials.db:                              ASCII text
./dosimetry/Radiotherapy/example12/data/LinacMaterials.db:                             ASCII text
./dosimetry/Radiotherapy/example2/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example3/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example3/data/PreciseMaterials.db:                            ASCII text
./dosimetry/Radiotherapy/example3/data/patient-HUmaterials.db:                         ASCII text
./dosimetry/Radiotherapy/example4/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example5/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example6/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example6/data/patient-HUmaterials.db:                         ASCII text
./dosimetry/Radiotherapy/example7/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example8/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example9/data/GateMaterials.db:                               ASCII text
./dosimetry/TEPCActor/data/myGateMaterials.db:                                         ASCII text
./dosimetry/UserSpectrum/data/GateMaterials.db:                                        ASCII text
./dosimetry/doseByRegions/data/AF_GateMaterials.db:                                    ASCII text
./dosimetry/doseactor/unvoxelized-example/data/GateMaterials.db:                       ASCII text
./dosimetry/doseactor/voxelized-phantom/data/GateMaterials.db:                         ASCII text
./dosimetry/dosimetry/brachytherapy/data/GateMaterials.db:                             ASCII text
./dosimetry/dosimetry/electron_radiotherapy/data/GateMaterials.db:                     ASCII text
./dosimetry/dosimetry/external-beam-therapy-photon/data/GateMaterials.db:              ASCII text
./dosimetry/dosimetry/molecular-therapy-I131/data/GateMaterials.db:                    ASCII text
./dosimetry/dosimetry/protontherapy/data/GateMaterials.db:                             ASCII text
./dosimetry/gaga-phsp/data/GateMaterials.db:                                           ASCII text
./imaging/CT/Fresnel_FFD/data/GateMaterials.db:                                        ASCII text
./imaging/CT/XRayRefraction/data/GateMaterials.db:                                     ASCII text
./imaging/CT/fixedForcedDetectionCT/data/ICRP_HUmaterials.db:                          ASCII text
./imaging/ComptonCamera/data/GateMaterials.db:                                         ASCII text
./imaging/DICOM/mhd-dcm-comparison/data/GateMaterials.db:                              ASCII text
./imaging/ExtendedVSource/CommonMacros/GateMaterials.db:                               ASCII text
./imaging/LUTDavisModel/GateMaterials.db:                                              ASCII text
./imaging/Optical/GateMaterials.db:                                                    ASCII text
./imaging/ProtonRadiography/data/GateMaterials.db:                                     ASCII text
./imaging/SPECT_FFD/data/GateMaterials.db:                                             ASCII text
./imaging/SPECT_GPU/GateMaterials.db:                                                  ASCII text
./imaging/Thermal/GateMaterials.db:                                                    ASCII text
./imaging/UserFluenceSource/data/GateMaterials.db:                                     ASCII text
./imaging/fastY90/data/GateMaterials.db:                                               ASCII text
./imaging/fluorescence/data/myGateMaterials.db:                                        ASCII text
./imaging/garf/data/GateMaterials.db:                                                  ASCII text
./imaging/garf/data/phantom_HUmaterials.db:                                            ASCII text
./imaging/vpgtle/stage0/data/GateMaterials.db:                                         ASCII text
./imaging/vpgtle/stage0/data/GateMaterialsElements.db:                                 ASCII text
./imaging/vpgtle/stage1/data/GateMaterials.db:                                         ASCII text
./imaging/vpgtle/stage1/data/patient-HUmaterials.db:                                   ASCII text
./imaging/vpgtle/stage2/data/GateMaterials.db:                                         ASCII text
./misc/GateMaterials.db:                                                               ASCII text
./misc/GateMaterialsGPU.db:                                                            ASCII text
./misc/TetrahedralMeshGeometry/data/GateMaterials.db:                                  ASCII text
./misc/TrackLengthFluenceExamples/TLFluence-Example1/Data/GateMaterials.db:            ASCII text
./misc/TrackLengthFluenceExamples/TLFluenceDistribution-Exampl1/Data/GateMaterials.db: ASCII text
./misc/geometry_STL/Suzanne/GateMaterials.db:                                          ASCII text
./misc/geometry_STL/kidneys/data/GateMaterials.db:                                     ASCII text
./misc/hybrid_volume_navigator/MergedVolume/GateMaterials.db:                          ASCII text
./misc/hybrid_volume_navigator/Reference/GateMaterials.db:                             ASCII text
SimonRit commented 3 years ago

I would also have added .db files to regular git. I'm not aware of any binary db file and I don't see any in the repo. Any clue how jm screwed up the repository with this .gitattributes file?

tbaudier commented 3 years ago

All these problems come from different commits. https://github.com/OpenGATE/GateContrib/commit/39e62ade3355b04fabaef08eaa204503d91b9133 includes .db files inside the LFS system (with a commit it's easy to remove). This commit https://github.com/OpenGATE/GateContrib/commit/32c47705ec400d779c56ba0e2d685b7173fbb27d includes a pdf file inside Github, so I need to rebase these commit to remove the Github history of this file and put it in the LFS. And same problem with .png files for this commit https://github.com/OpenGATE/GateContrib/commit/d1b6993556e07f796e0bdcba6d174d4d2cc8a1a6

SimonRit commented 3 years ago

39e62ad includes .db files inside the LFS system (with a commit it's easy to remove).

With a clear commit message explaining why...

This commit 32c4770 includes a pdf file inside Github, so I need to rebase these commit to remove the Github history of this file and put it in the LFS. And same problem with .png files for this commit d1b6993

I think @dsarrut chose consciously to include some small binary files in Gate repositories.

tbaudier commented 3 years ago

I rewrote the history for these 3 previous commits with a rebase. The updated repository can be found here: https://gitlab.in2p3.fr/tbaudier/gatecontrib/-/commits/master And now the initial LFS error is not present.

But I saw that some binary are not staged by LFS (not present in gitattribute):

GATE-RTion/DoseActor/DoseActor_Validation sheet_2018-06-19.xlsx
GATE-RTion/DoseActor2-CT/template_results.ods
GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.odt
dosimetry/Radiotherapy/example11/data/1000p.IAEAphsp
tbaudier commented 3 years ago

I updated the repository to remove the previous binary files from the history and put them in LFS. Everything can be seen here: https://gitlab.in2p3.fr/tbaudier/gatecontrib I had also update the readme to announce the rebase. Is it OK, Can I pull inside this github repository ?

SimonRit commented 3 years ago

I was under the impression that with git lfs, we should have a small .git directory, with binary files being only pointers to files hosted on github. When I clone your repo, the .git directory is 686 MB. I don't understand what the problem is but when I run the command suggested here, with the title A blazingly fast shell one-liner, it seems that some large commits are still present but I don't understand how to clean this up. The commits which are large seem inaccessible so here stops my understanding of git. (When I run git clone --single-branch, it's worse, 952 MB!)

tbaudier commented 3 years ago

Thank you for the remarks. On my computer, I can see different things:

  1. If I clone the gitlab repository (git clone), the total size is 2.6 GB with 1.6 GB for .git folder
  2. If I clone the gitlab repository without pulling LFS (GIT_LFS_SKIP_SMUDGE=1 git clone), the total size of the repository is 694 MB with 686 MB for .git folder
  3. If I clone the master branch only of the gitlab repository without pulling LFS (GIT_LFS_SKIP_SMUDGE=1 git clone --single-branch), the total size is 28 MB with 20 MB for .git folder
  4. If I clone the master branch only of the gitlab repository (git clone --single-branch), the total size is 1.9 GB with 952 MB for .git folder

For the point 1/ and 2/, the size is important because I cloned all the branches (master, revert and testRebase) and the two last branches could be erased. They still contains binary files. For the 3/ and 4/ points the size is less important, but we can still encounter old large files in the history. For example with this commit converted to LFS now or this one deleted now.

To clean the repository, I need to remove all binary files in the history. Your link could help me but:

I will try to propose something using all these tools.

SimonRit commented 3 years ago 
3\. If I clone the master branch only of the gitlab repository without pulling LFS (`GIT_LFS_SKIP_SMUDGE=1 git clone --single-branch`), the total size is 28 MB with 20 MB for `.git` folder

I had tried git clone --single-branch but without GIT_LFS_SKIP_SMUDGE. This is much better thanks. I would list all files in the history and check those we want to keep / remove the others with bfg. For example, the three following commands halved the size of the repo (19MB -> 9MB):

java -jar ~/Downloads/bfg-1.13.0.jar --delete-files SJAN_GC_2009.ppt
java -jar ~/Downloads/bfg-1.13.0.jar --delete-folders source
git reflog expire --expire=now --all && git gc --prune=now --aggressive
tbaudier commented 3 years ago

I identified 92 files with a size superior to 1 kB in the history. Some of them could be text file (I do not know how to check that). This is the list:

2c1c2236fed9  1.2KiB cluster_tools/jobsplitter/script/condor.script
0587e5902bd1  1.3KiB source/externals/itk-mhd/itkzlib/INDEX
0e06337e4b41  1.7KiB cluster_tools/jobsplitter/script/openPBS.script
24271f8b8241  2.0KiB source/externals/itk-mhd/itkzlib/zlib.def
80a539348503  2.2KiB gc/gc.aux
414b4fc72a91  2.3KiB GateCommon.gmk
625d893844ab  2.8KiB benchmarks/benchHadron/proton-dose/output/plot.gp
d392bf934bac  4.6KiB gc/iopart12.clo
dec57470037f  4.6KiB gc/iopart10.clo
257a79ea52b5  5.0KiB source/physics/src/GateHadronElasticPB.cc.old
b5d11b249ce2  5.3KiB source/geometry/include/GateDMapoperators.ihh
112d2f1fc3b2  5.8KiB examples/example_DNA/output/Target2.root
d4ebc478ad28  5.9KiB examples/example_DNA/output/Target3.root
ca07da48fdc6  5.9KiB examples/example_DNA/output/Target1.root
b02108f0f8c3  6.3KiB MacroFile_Example/macrosSystem/PET_acquis.in
4f026501af91  6.4KiB MacroFile_Example/macrosMove/PET_acquis_rotation.in
b4fee9efdd1f  6.6KiB MacroFile_Example/macrosMove/PET_acquis_translation.in
35e1eeb5b7be  6.6KiB MacroFile_Example/macrosMove/PET_acquis_osctransl.in
a175ed557f29  7.2KiB MacroFile_Example/macrosMove/PET_acquis_orbiting.in
c45e6be243a8  7.2KiB MacroFile_Example/macrosMove/PET_acquis_eccentRotation.in
6208db87a7ab  7.8KiB source/digits_hits/src/GateLETActor.cc.save.1
b70f393b778e   10KiB source/general/include/GateImage.hh.orig
5799920364ea   11KiB env_gate.csh
94fad4dead03   11KiB dosimetry/EMFields/mac/main.submit
a7c4d9ab21a6   13KiB example_UserSpectrum/output/SpectrumOutput.png
b2e0f5cf13ae   16KiB benchmarks/benchDNA/Target2.gif
739616e0e6e8   16KiB benchmarks/benchDNA/Target1.gif
748815fb0f3f   18KiB benchmarks/benchDNA/Target.gif
e584ccd0b345   19KiB example_fastY90/output/fastY90output.png
db4a2e96e591   21KiB gc/harvard.perl
b08b8f68c17c   21KiB gc/figures/1GyFullPET.jpg
5b4992d9f7d9   23KiB gc/figures/prof_C11_O15_10Gy_v1.jpg
4d116237407a   24KiB gc/jphysicsB.bst
bbcd6059173c   24KiB gc/kluwer.bst
1c761698c4a6   25KiB gc/figures/dfo.jpg
f53512a5d7ac   25KiB gc/figures/histo_lung_O15_paper.jpg
b754096b964d   25KiB gc/figures/prof_C11_O15_10Gy_v3.jpg
2aa8cc94cd1a   25KiB gc/apsr.bst
1a66c4c49638   25KiB gc/figures/histo_lung_C11_paper.jpg
9ebcc9c76413   25KiB gc/agsm.bst
1c26a913ffa0   25KiB gc/dcu.bst
aa19dd2f36fd   25KiB gc/figures/prof_gaussPET_C11_v1.jpg
2f8259f877d9   25KiB gc/jmr.bst
dc3b6e5c4274   26KiB source/digits_hits/src/GatePhaseSpaceActor.cc.orig
3ebfa0ef5b5f   26KiB gc/figures/10-5-1-Gy.jpg
3ad344ed6e5a   33KiB gc/figures/prof_C11_O15_10Gy_v2.jpg
a9d12ea2566a   37KiB gc/iopart.cls
c6c7b8dd872e   37KiB gc/nederlands.bst
f04370338fb9   38KiB GATE-RTion/DoseActor2-CT/CTvalidation.ods
3a14d626d82f   40KiB examples/example_doseactor/voxelized-phantom/data/PhantomSlice.mha
d71b00b74822   44KiB gc/figures/C11_1Gy_v1.jpg
8b9496606702   45KiB gc/figures/C11_1Gy_v2.jpg
f2c01bb9dfae   46KiB gc/figures/O15_5Gy_v1.jpg
71d85df90ce8   46KiB gc/figures/C11_5Gy_v1.jpg
840381550bcb   47KiB gc/figures/C11_10Gy_v1.jpg
285c513ac054   47KiB misc/geometry_STL/Suzanne/Suzanne.stl
b71437b5681b   49KiB gc/figures/C11_O15_10Gy_v1.jpg
61f6bf3c8a0f   49KiB gc/figures/gaussPET_C11_v2.jpg
e0a808f43133   50KiB gc/figures/O15_10Gy_v1.jpg
bd6b3c17ac2b   51KiB gc/figures/C11_10Gy_v2.jpg
a550f465a6a0   60KiB gc/figures/O15_1Gy_v1.jpg
d096f6750bc3   63KiB gc/figures/1GyFullPET_v1.jpg
75209c9c8e50   67KiB misc/geometry_STL/kidneys/data/Label94.stl
502c3135caba   79KiB source/test.xml.bak
ac316c10d6fe   89KiB misc/geometry_STL/kidneys/data/Label91.stl
a492126ae26e   93KiB gc/figures/bloc_detec_HR.jpg
237bb18d6cb6  115KiB gc/figures/phantomCT.jpg
babc1cc72651  121KiB gc/figures/HR_Simu_Gate_GC.jpg
ec4a9b51b6cb  124KiB gc/figures/C11_O15_10Gy_v2.jpg
1f86319e755d  128KiB gc/figures/gaussPET_C11_v1.jpg
1c4dd29669af  129KiB gc/figures/gaussPET_O15_v1.jpg
fa319773c090  146KiB gc/figures/3D-Dose_poumon.jpg
57e89e2a2470  151KiB gc/figures/3D-PETC11_poumon.jpg
2ae2a4579c16  171KiB gc/figures/3D-PETO15_poumon.jpg
aedd22491cec  175KiB gc/figures/sagital-Dose.avi
bf1499775911  218KiB gc/figures/coronal-Dose.avi
37683cc4badd  235KiB misc/geometry_STL/kidneys_STL.png
97196eaeb112  272KiB misc/geometry_STL/kidneys/data/Label93.stl
cb53268d351b  283KiB misc/geometry_STL/kidneys/data/Label90.stl
1fbb95d3a9e2  291KiB gc/figures/axial-Dose.avi
7cf597ea2cfe  293KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.docx
24efa59cc94d  319KiB gc/figures/coronalPoumon.avi
10765e1190cc  329KiB gc/figures/sagitalPoumon.avi
e35eb2277a9e  354KiB gc/figures/axialPoumon.avi
0f61bf3a4842  371KiB misc/geometry_STL/kidneys/data/Label92.stl
149890b0ee0c  400KiB misc/geometry_STL/kidneys/data/Label89.stl
38080cca9e7c  487KiB GATE-RTion/TPSPencilBeamSource/results/TPS-Validation-Summary.gnumeric
0d27562c8145  617KiB GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.pdf
d7fdd2297cbf  618KiB GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.pdf
84d208aa3f1d  714KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.pdf
fa3ac249a3cc  889KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.docx
0ca6e3c60a3f  927KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.pdf

To decrease the size, I suggest to remove all links to the gcand source old folders. And then, remove the links to the following 37 files. What do you think? We can use BFG like you do or a git-filter-branch (what I used for the previous rebase)

2c1c2236fed9  1.2KiB cluster_tools/jobsplitter/script/condor.script
0e06337e4b41  1.7KiB cluster_tools/jobsplitter/script/openPBS.script
414b4fc72a91  2.3KiB GateCommon.gmk
625d893844ab  2.8KiB benchmarks/benchHadron/proton-dose/output/plot.gp
112d2f1fc3b2  5.8KiB examples/example_DNA/output/Target2.root
d4ebc478ad28  5.9KiB examples/example_DNA/output/Target3.root
ca07da48fdc6  5.9KiB examples/example_DNA/output/Target1.root
b02108f0f8c3  6.3KiB MacroFile_Example/macrosSystem/PET_acquis.in
4f026501af91  6.4KiB MacroFile_Example/macrosMove/PET_acquis_rotation.in
b4fee9efdd1f  6.6KiB MacroFile_Example/macrosMove/PET_acquis_translation.in
35e1eeb5b7be  6.6KiB MacroFile_Example/macrosMove/PET_acquis_osctransl.in
a175ed557f29  7.2KiB MacroFile_Example/macrosMove/PET_acquis_orbiting.in
c45e6be243a8  7.2KiB MacroFile_Example/macrosMove/PET_acquis_eccentRotation.in
5799920364ea   11KiB env_gate.csh
94fad4dead03   11KiB dosimetry/EMFields/mac/main.submit
a7c4d9ab21a6   13KiB example_UserSpectrum/output/SpectrumOutput.png
b2e0f5cf13ae   16KiB benchmarks/benchDNA/Target2.gif
739616e0e6e8   16KiB benchmarks/benchDNA/Target1.gif
748815fb0f3f   18KiB benchmarks/benchDNA/Target.gif
e584ccd0b345   19KiB example_fastY90/output/fastY90output.png
f04370338fb9   38KiB GATE-RTion/DoseActor2-CT/CTvalidation.ods
3a14d626d82f   40KiB examples/example_doseactor/voxelized-phantom/data/PhantomSlice.mha
285c513ac054   47KiB misc/geometry_STL/Suzanne/Suzanne.stl
75209c9c8e50   67KiB misc/geometry_STL/kidneys/data/Label94.stl
ac316c10d6fe   89KiB misc/geometry_STL/kidneys/data/Label91.stl
37683cc4badd  235KiB misc/geometry_STL/kidneys_STL.png
97196eaeb112  272KiB misc/geometry_STL/kidneys/data/Label93.stl
cb53268d351b  283KiB misc/geometry_STL/kidneys/data/Label90.stl
7cf597ea2cfe  293KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.docx
0f61bf3a4842  371KiB misc/geometry_STL/kidneys/data/Label92.stl
149890b0ee0c  400KiB misc/geometry_STL/kidneys/data/Label89.stl
38080cca9e7c  487KiB GATE-RTion/TPSPencilBeamSource/results/TPS-Validation-Summary.gnumeric
0d27562c8145  617KiB GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.pdf
d7fdd2297cbf  618KiB GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.pdf
84d208aa3f1d  714KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.pdf
fa3ac249a3cc  889KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.docx
0ca6e3c60a3f  927KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.pdf
SimonRit commented 3 years ago

Thanks Thomas for the huge effort. I had another look and yes, I'm in favor of removing the folders sources (source files of Gate) and gc (draft article of S. Jan, T. Frisson and @dsarrut). I did not follow the creation of GateContrib but I'm guessing that the motivation for starting from the original Gate repository was to keep the history of the relevant files. These two subfolders are not relevant for GateContrib and can be removed IMHO. I would also remove the root folders cluster_tools, cmake-modules and superbuild for the same reasons. The history of these files will be kept in the Gate repository. I trust you for the other files but I can have another last look if you'd like.

dsarrut commented 3 years ago

exactly: for sure you can remove 'gc', 'sources', 'cluster_tools', 'cmake-modules' and'superbuild'. Also, remove: 'MacroFile_Example', 'benchmarks' I think. thanks !

tbaudier commented 3 years ago

So I deleted all that old folders:

I also removed that folders:

I also remove the old links to that binary files:

414b4fc72a91  2.3KiB GateCommon.gmk
5799920364ea   11KiB env_gate.csh
285c513ac054   47KiB misc/geometry_STL/Suzanne/Suzanne.stl
37683cc4badd  235KiB misc/geometry_STL/kidneys_STL.png
75209c9c8e50   67KiB misc/geometry_STL/kidneys/data/Label94.stl
ac316c10d6fe   89KiB misc/geometry_STL/kidneys/data/Label91.stl
97196eaeb112  272KiB misc/geometry_STL/kidneys/data/Label93.stl
cb53268d351b  283KiB misc/geometry_STL/kidneys/data/Label90.stl
0f61bf3a4842  371KiB misc/geometry_STL/kidneys/data/Label92.stl
149890b0ee0c  400KiB misc/geometry_STL/kidneys/data/Label89.stl
f04370338fb9   38KiB GATE-RTion/DoseActor2-CT/CTvalidation.ods
7cf597ea2cfe  293KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.docx
38080cca9e7c  487KiB GATE-RTion/TPSPencilBeamSource/results/TPS-Validation-Summary.gnumeric
d7fdd2297cbf  618KiB GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.pdf
84d208aa3f1d  714KiB GATE-RTion/DoseActor2-CT/Voxelized-geometry_Validation_Procedure_2019-08-20.pdf
fa3ac249a3cc  889KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.docx
0ca6e3c60a3f  927KiB GATE-RTion/DoseActor/DoseActor_Validation_Procedure_2018-06-19.pdf

Some of them were previously removed, some still present as LFS pointer. In such a case I needed to redo the pointer with that commit. The new branch can be seen here

And concerning the size of the folder:

If it's OK for you I can force the push in the main repository. I will also remove the revertbranch. @dsarrut can I remove the json-test branch ?

tbaudier commented 3 years ago

For the 3 current PR:

SimonRit commented 3 years ago

Thanks. I guess #22 should be checked before merging. #49 needs to be validated by @dsarrut (whether it's the correct documentation file and whether it should be committed here). After some last checks, I think GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db, the largest file in the repo (800 KB), should be removed since it can be generated (I guess?). That may be the reason why .db files had been added to .gitattributes. And I think that all stl files which are not lfs files should be removed. In any case, your current branch looks much better. You can force push, that's the best way to proceed in my limited git knowledge.

tbaudier commented 3 years ago

Thanks. I guess #22 should be checked before merging. #49 needs to be validated by @dsarrut (whether it's the correct documentation file and whether it should be committed here).

I do not want to merge these PR, just rebase them to be able to merge them after review.

After some last checks, I think GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db, the largest file in the repo (800 KB), should be removed since it can be generated (I guess?). That may be the reason why .db files had been added to .gitattributes.

I do not have any idea and it represents only 0.04 % of the repository

And I think that all stl files which are not lfs files should be removed. In any case, your current branch looks much better. You can force push, that's the best way to proceed in my limited git knowledge.

You're right, I did not check the size of files after rebasing everything but it seems some .stl files "appeared". So I remove the old links:

75209c9c8e50   67KiB misc/geometry_STL/data/Label94.stl
ac316c10d6fe   89KiB misc/geometry_STL/data/Label91.stl
97196eaeb112  272KiB misc/geometry_STL/data/Label93.stl
cb53268d351b  283KiB misc/geometry_STL/data/Label90.stl
0f61bf3a4842  371KiB misc/geometry_STL/data/Label92.stl
149890b0ee0c  400KiB misc/geometry_STL/data/Label89.stl
SimonRit commented 3 years ago

I do not have any idea and it represents only 0.04 % of the repository

You're looking at the lfs version I guess (the current version), so I was not clear enough. The problem is with

e08c2398e8e6 790KiB GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db

(using your gitlab repo). Check the history of the file to clarify what happened... The problem is in commit d317909cb7b1223e383b24ecca83c1524aa9d35a (github version). The file has 26 K lines and represents probably more than 10% of the repo... Removing the first version will solve the problem.

dsarrut commented 3 years ago

I am not sure that this file 'GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db' should/could be removed, any idea @djboersma ?

SimonRit commented 3 years ago

At least the non lfs version (only visible in the history) of it should be removed in my opinion. And this file does not seem to be used in any macro?

tbaudier commented 3 years ago

I do not understand what you mean by "non LFS version". GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db is not stored in LFS. We cannot see it with: git lfs ls-files

SimonRit commented 3 years ago

Sorry, I have been fooled by the commit messages I guess (and by your strange 0.04% calculation, 792KB/8MB~=10%). It's a pity it's not straightforward to identify lfs files on github (gitlab seems more convenient). Apologies for the noise anyway. The questions now are: is this file needed (@djboersma, @GrevillotL)? If yes, lfs or not?

tbaudier commented 3 years ago

No problem. I have the same point of view about the identification of LFS files in Github vs. Gitlab. And for the 0.04%, I took the whole directory (=800K/1.9G). For LFS for text file, it seems that it's not obvious (https://stackoverflow.com/questions/40795130/how-can-i-track-text-files-with-proper-line-endings-normalization-using-git-lfs)

SimonRit commented 3 years ago

Thanks for the pointer. It seems to be a Windows issue but not lfs then and only one question left on my side:

is this file needed (@djboersma, @GrevillotL)?

djboersma commented 3 years ago

Thanks for the pointer. It seems to be a Windows issue but not lfs then and only one question left on my side:

is this file needed (@djboersma, @GrevillotL)?

The GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db file looks like output from the HounsfieldMaterialGenerator and indeed it doesn't seem to be used in any of the macros in the corresponding mac directory. I agree that it can be removed, we probably forgot to clean it up before uploading these data to GitHub.

GrevillotL commented 3 years ago

Hello, I indeed think this file is not needed – Marta can you confirm??

Best, Loïc

From: David Boersma [mailto:notifications@github.com] Sent: Donnerstag, 15. Oktober 2020 15:09 To: OpenGATE/GateContrib GateContrib@noreply.github.com Cc: Grevillot Loic loic.grevillot@medaustron.at; Mention mention@noreply.github.com Subject: Re: [OpenGATE/GateContrib] LFS data (#50)

Thanks for the pointer. It seems to be a Windows issue but not lfs then and only one question left on my side:

is this file needed (@djboersmahttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdjboersma&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=r2y%2F0iFs3uV3vMpjc2KhW%2FpqtiX1f0GUfaZGZcZzOY0%3D&reserved=0, @GrevillotLhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGrevillotL&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=2%2FrODsI7bH8ALIYXdeLJDVQeGhMlhKRmRyrTFg7ZxuY%3D&reserved=0)?

The GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db file looks like output from the HounsfieldMaterialGenerator and indeed it doesn't seem to be used in any of the macros in the corresponding mac directory. I agree that it can be removed, we probably forgot to clean it up before uploading these data to GitHub.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenGATE%2FGateContrib%2Fissues%2F50%23issuecomment-709312880&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=%2BXZXTRp5dxG6U2c%2FxVUKhFS%2B8wqB%2FFeKb4IZOD13WLw%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFPJCONEIHQWIXZX2BXLAWTSK3X5FANCNFSM4SIOU6FQ&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166016155&sdata=ze1K5UnBpqdTw2W0Aa3CnC%2BgUJaDH6WhJXd%2BpAVTIFg%3D&reserved=0. Disclaimer: Please notice our E-Mail Disclaimer http://www.medaustron.at/email-disclaimer/

GrevillotL commented 3 years ago

Hello,

The file is generated and called back with the commands:

/gate/HounsfieldMaterialGenerator/SetOutputMaterialDatabaseFilename data/{tagName}-HUmaterials.db

/gate/geometry/setMaterialDatabase data/{tagName}-HUmaterials.db

I don’t know what you are talking about ☺ but if you need the exact same material data base for all of your simulations (i.e. same density tolerance and look up tables) you can also generate the file once, comment the generator command line and re-use the file.

Cheers, Marta

From: Grevillot Loic Sent: 15 October 2020 16:43 To: OpenGATE/GateContrib reply@reply.github.com; OpenGATE/GateContrib GateContrib@noreply.github.com Cc: Mention mention@noreply.github.com; Bolsa-Ferruz Marta marta.bolsa-ferruz@medaustron.at Subject: RE: [OpenGATE/GateContrib] LFS data (#50)

Hello, I indeed think this file is not needed – Marta can you confirm??

Best, Loïc

From: David Boersma [mailto:notifications@github.com] Sent: Donnerstag, 15. Oktober 2020 15:09 To: OpenGATE/GateContrib GateContrib@noreply.github.com<mailto:GateContrib@noreply.github.com> Cc: Grevillot Loic loic.grevillot@medaustron.at<mailto:loic.grevillot@medaustron.at>; Mention mention@noreply.github.com<mailto:mention@noreply.github.com> Subject: Re: [OpenGATE/GateContrib] LFS data (#50)

Thanks for the pointer. It seems to be a Windows issue but not lfs then and only one question left on my side:

is this file needed (@djboersmahttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdjboersma&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=r2y%2F0iFs3uV3vMpjc2KhW%2FpqtiX1f0GUfaZGZcZzOY0%3D&reserved=0, @GrevillotLhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGrevillotL&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=2%2FrODsI7bH8ALIYXdeLJDVQeGhMlhKRmRyrTFg7ZxuY%3D&reserved=0)?

The GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db file looks like output from the HounsfieldMaterialGenerator and indeed it doesn't seem to be used in any of the macros in the corresponding mac directory. I agree that it can be removed, we probably forgot to clean it up before uploading these data to GitHub.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenGATE%2FGateContrib%2Fissues%2F50%23issuecomment-709312880&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166006203&sdata=%2BXZXTRp5dxG6U2c%2FxVUKhFS%2B8wqB%2FFeKb4IZOD13WLw%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFPJCONEIHQWIXZX2BXLAWTSK3X5FANCNFSM4SIOU6FQ&data=02%7C01%7Cloic.grevillot%40medaustron.at%7Cab505dcb2b6b47f209bf08d8710b6c78%7Cadb749e06dfd4f6fbea8cd9f8e0c9b89%7C0%7C0%7C637383641166016155&sdata=ze1K5UnBpqdTw2W0Aa3CnC%2BgUJaDH6WhJXd%2BpAVTIFg%3D&reserved=0. Disclaimer: Please notice our E-Mail Disclaimer http://www.medaustron.at/email-disclaimer/

tbaudier commented 3 years ago

Thank you all, I delete the file and push everything

SimonRit commented 3 years ago

/gate/HounsfieldMaterialGenerator/SetOutputMaterialDatabaseFilename data/{tagName}-HUmaterials.db /gate/geometry/setMaterialDatabase data/{tagName}-HUmaterials.db

I can't find any macro with these commands... Where is it?

Thank you all, I delete the file and push everything

Thanks.

tbaudier commented 3 years ago

The push is done

tbaudier commented 3 years ago

I pushed 3 branches related to the 3 PR. I cherry-picked the new commit. I cannot push directly on your local github to update the PR automatically because I do not have rights.

SimonRit commented 3 years ago

47 and #49 are ready to be merged

tbaudier commented 3 years ago

Thank you I merged