git lfs settings need to be tweaked

[serbinsh]

@JulienLamour we should not be blanket saving all .csv files as LFS, as this then also includes small files like "Description.csv" files. This is creating extra overhead issues that make code management harder

I suggest two options: 1) a separate data repo specific to this effort that is used to store the datasets or 2) change the Description files from csv to txt so they arent included in the LFS. Or alternatively, we have to manually curate each required LFS file

[serbinsh]

That said - I see now that we need to be clear that we need to run

git lfs pull

to sync anything saved as LFS locally

[serbinsh]

But got a bunch of errors

lee-169142:Burnett_et_al_2021 sserbin$ git lfs pull
[d791ec1a83017c30201124615a77c08a87fa07e254dc2d7ea64e34e9ede5a5cb] Object does not exist on the server: [404] Object does not exist on the server
[22160b2034984fa9df4136e11ed85886e1fa1c36ebc1cc0a43be9099d67369fc] Object does not exist on the server: [404] Object does not exist on the server
[7fc4a0c9b8936ba13306521501743e3a217a42c6e8c252b83bee99ec977599ce] Object does not exist on the server: [404] Object does not exist on the server
[b8890f9c44780179a26d5bbecabbdbbbe79d6a7f597990119df0570d0193135e] Object does not exist on the server: [404] Object does not exist on the server
[51f692fdd7b82b5cb251d7d424a7394510d0e62901ec7e4ea75e1a4b46193c98] Object does not exist on the server: [404] Object does not exist on the server
[2853d6f61c4678f3b1fda1ecbd1b6dce83686b39a85ec673f1b174eec1e79c4e] Object does not exist on the server: [404] Object does not exist on the server
[b0b1a4a234939c6f1624523c6c7bf81d966d68990ab0175aa2fa3ebb02870ad6] Object does not exist on the server: [404] Object does not exist on the server
[592877a386b3b29f20415c05d57f8a18881e519f631fe8b0cb3bcb28195d5f61] Object does not exist on the server: [404] Object does not exist on the server
[f8be8b84917e44b1f3e6296e921595391e5bc551dfdd5acde46ed39001268600] Object does not exist on the server: [404] Object does not exist on the server
[21dc51d65546260ec1ac16a7178e13315cebb1d916249ab2fca836d7f9405de4] Object does not exist on the server: [404] Object does not exist on the server
[290ee6b51f7aeea6f6a290d30ea3f864dd3dde93a179973e3af8e5e08f14e52c] Object does not exist on the server: [404] Object does not exist on the server
[7fa6278f7a9ea366fd897256729574659b4f1ca3dbb3edcb1b254a9ea56033d1] Object does not exist on the server: [404] Object does not exist on the server
[0386d22f457e1c1209cd2252cf2db978a9384376110290ce4aa1cfb9594914a3] Object does not exist on the server: [404] Object does not exist on the server
[41271351a83c1b2724a23c5e01931f3e389af4aceb32053f985aef271b17b8c5] Object does not exist on the server: [404] Object does not exist on the server
[d36aafdf25aa6185d6964b68373a7ebd889bf07fdce84ce6ecd3a10fbd321731] Object does not exist on the server: [404] Object does not exist on the server
[7db9c9e48780cd23ba074cbb19cc98f11661fd640591661d8a96a575a480f3c0] Object does not exist on the server: [404] Object does not exist on the server
[45cd990029f292de9cf27ab64759e8c7acdd5434db5310e3b0a4cec491957f20] Object does not exist on the server: [404] Object does not exist on the server
[ec3280e9cc1d961f884889fd24ada16ac4f99f3acf0dc7b8c76a2dc824d24445] Object does not exist on the server: [404] Object does not exist on the server
[245160418416a645aeb0d546460a3ce56cdec508c52db62004ba98976ca5dbd3] Object does not exist on the server: [404] Object does not exist on the server
[def419b1ea1d0f26ed326e88800c0e705c5a1cdf68585fc8bb40a64dccc2e8c9] Object does not exist on the server: [404] Object does not exist on the server
[4bc9353807e740d27a69b9a9cd317eb225b540a7c1c32230f92150f2f081379e] Object does not exist on the server: [404] Object does not exist on the server
[f88d6862787cb2904655b61c75aa01a218ad03bda0476014977c1a39b8abbfe3] Object does not exist on the server: [404] Object does not exist on the server
[5f338dba8301df80bbe452a133d9063344b7d9001fc86d5ad4c03bebd2cda3b8] Object does not exist on the server: [404] Object does not exist on the server
[dcd71f981aacb53dcc51665df2084e9cea35f7bf0f618fdb67f7488a17a835d4] Object does not exist on the server: [404] Object does not exist on the server
[d2e768fbec89a4664c6e5da2fe56f3518fe4e1ba63e896c6feda15c2a3f2d9db] Object does not exist on the server: [404] Object does not exist on the server
[e1b4d160304452030f3a77f27eccc9fb1c547155e229ea8533be4a3cfca97dab] Object does not exist on the server: [404] Object does not exist on the server
[e57fb7743ba4d5a80e0991e65da4547926743ca677c97bda57b17861ad2d39a8] Object does not exist on the server: [404] Object does not exist on the server
[305df9a139608ed1fe55df6fdc2d1e82d47e25a58629e9479d5c115802a4c4b1] Object does not exist on the server: [404] Object does not exist on the server
[143a8a97279fe7fb676a8d5d1b92bd411d451155f6febd666924c138f38e6888] Object does not exist on the server: [404] Object does not exist on the server
[0975e1029fd1b766706797bc1c26df9b5caf14be546dc3959afd1db3c30b758e] Object does not exist on the server: [404] Object does not exist on the server
[7c58377a82a4fbb0282323490fa0126984be7e3bd72892fbd33a99cc3a2cdfa5] Object does not exist on the server: [404] Object does not exist on the server
[0dd5791864fa75aedf1721ad737cbfca02039ae3df4667659d45c373cc272441] Object does not exist on the server: [404] Object does not exist on the server
[854ee53bbc5a8ae79b3493bd0a18d41de7b33b2c3e462dc4e6791d365bb3d805] Object does not exist on the server: [404] Object does not exist on the server
[5af361f558a0dc648d19d658dbe6d3833d717c7bed1aa755e1fbdb847a089344] Object does not exist on the server: [404] Object does not exist on the server
[139086e2305a6466d24dbad5824996fa93b6183cc22e8f07c99bda099e3070c1] Object does not exist on the server: [404] Object does not exist on the server
[53faa3de6a9e83820bd658b8da729d3714bae70b89b084c3178cd5f216026c42] Object does not exist on the server: [404] Object does not exist on the server
[6c926ed6f38045578a45454f60e6e88368a03d9db0f946bfec7619649db14768] Object does not exist on the server: [404] Object does not exist on the server
[2719525f01bcbd715a9782a650c18eb68e6ea196295ac784d916f3af1fbe5582] Object does not exist on the server: [404] Object does not exist on the server
[fef29d546c0afc9db986d758588ed721aeae1a06e07b7295d1f90a6b1784d8b9] Object does not exist on the server: [404] Object does not exist on the server
[bb542336765d5852c1a96bedebf13cfa9351079e7445bc86783467082da786dc] Object does not exist on the server: [404] Object does not exist on the server
[a1372bf2ca9eed3be50a0b99eb9dabfeb0172e475f461cc06cad98c263c4b2e0] Object does not exist on the server: [404] Object does not exist on the server
[a46cd8b8188f35956604f9af50b58673b9b79d77eb4f6ef97488b3c23b3d3297] Object does not exist on the server: [404] Object does not exist on the server
[8e70441d9643d7e27e11c0393287eddf44488e2679462f5018b7580b7f1fe7ad] Object does not exist on the server: [404] Object does not exist on the server
[21664e958f3f5a96c3af7a51958891e6a95c9e62bf9a2f1b6817024328aa49b7] Object does not exist on the server: [404] Object does not exist on the server
[daad2a57bfc8a2a17f4f41fe0191b93168a3dd42f65417a7e4a78a93e53bd91a] Object does not exist on the server: [404] Object does not exist on the server
[04da96f076b4fdf9c7b79c8693b123a3e12eee25f7002047d0e2fab6d8fc8daa] Object does not exist on the server: [404] Object does not exist on the server
[ac3c45356878e514a568d98330e0d586094c3df61afd82e47b91dc24acea1adf] Object does not exist on the server: [404] Object does not exist on the server
[87b620d42896990bcb41aff70be547aa71c03814c217d7aadc52a72e8422e5ef] Object does not exist on the server: [404] Object does not exist on the server
[fccff78506ed38a84b53d13c13a6c513061aa1bcbaca105de7a6b6b57674c060] Object does not exist on the server: [404] Object does not exist on the server
[1ab74b3ae81912eb06313c690f9eb9b5feb41088f628dad4397abcb97de3f9dc] Object does not exist on the server: [404] Object does not exist on the server
[1cf9efd423c447987c213a5967d38993b86e79d080ac747b4bc05b4b3e65bd40] Object does not exist on the server: [404] Object does not exist on the server
[d8b49bd8350b1438eb89f1be48e3ace678beec338fbc97e3b096f486cc5188b4] Object does not exist on the server: [404] Object does not exist on the server
[ebad1e595d2b3e4dd28016cdda9bef7f86b433b4f98ea0a23eeb73149e07f1e4] Object does not exist on the server: [404] Object does not exist on the server
[bc924ebe088085e47c64163172d28a3729147394c26d67d6310e8866ad59f460] Object does not exist on the server: [404] Object does not exist on the server
[d3ac283243ee20aed5cbc84e0c2444a4d6e0d3124a0061bc7a8bf8238fac55b0] Object does not exist on the server: [404] Object does not exist on the server
[4f20b562cfb542ace1752640e290fc08fcde5e6ce2a18957c663c0575833bd1a] Object does not exist on the server: [404] Object does not exist on the server
[aaa45c3f0521920855fe534556939404728d7c7befc470a016e893806462f179] Object does not exist on the server: [404] Object does not exist on the server
[b6f1079adc25e72eaaf55f9906afcf4f17557bd76c2c3bd68c1acd276263cc20] Object does not exist on the server: [404] Object does not exist on the server
[c0667cb33cced3d6be65aacbf5f7c1bd4f8a35a373e094df470bc1ba99a62603] Object does not exist on the server: [404] Object does not exist on the server
[a8c1e73bed346017b4b7ed522fdc2fffb0b6b36ad6f06b374deb7861fb7f927a] Object does not exist on the server: [404] Object does not exist on the server
[b74d8decac9f80b08d789f447aafd89d5be731a9e02ce169273d68e65335f594] Object does not exist on the server: [404] Object does not exist on the server
[92323a3bb96dbc1b0e7eff627c11791935965000f835e1abb09b4efb53054d8f] Object does not exist on the server: [404] Object does not exist on the server
[0ae7f623a7594629d6777d7dfcddb1a86b070f9731461f60d97284b686715fc2] Object does not exist on the server: [404] Object does not exist on the server
[775a96abca742d33cc270a3f10dfa3118b83c51fa2879d5d567630b8744a439e] Object does not exist on the server: [404] Object does not exist on the server
[44e512f27ba13068e41cde268f685cf6a453747955df3929b186c48b06e5285f] Object does not exist on the server: [404] Object does not exist on the server
[f20bae109d5ee7343bdfb644e5e276d8c67c690a652ec1112fde3b3bbbc87787] Object does not exist on the server: [404] Object does not exist on the server
[1478924dd9da32972bf63972a4d24dc09ddb09fc1788d03599d3a70d1ea72678] Object does not exist on the server: [404] Object does not exist on the server
[d903714922746e98c9e3deef96f33caca9b2ccc42cc7a96b0d07396be46b7414] Object does not exist on the server: [404] Object does not exist on the server
[4e90d143aac4162d849137cd52bda7b25916b1435ae68ac2da0ac4e89a956f50] Object does not exist on the server: [404] Object does not exist on the server
[5fbb490fc85eb3cd05d36b24fe09c227c7b6159d0645c75324acfc8360a2dd59] Object does not exist on the server: [404] Object does not exist on the server
[7e14afed45d30052a5d19f77897f65f105960d19c26539cab0892c3ad5b91cfe] Object does not exist on the server: [404] Object does not exist on the server
[c307efe33b0a859179b752713d8daad9e6659cbe91a067c34cf7202c0ff6b49d] Object does not exist on the server: [404] Object does not exist on the server
[7c7cf9b944993f94a12ad97f41ce30a4405297964e8ee80cca172d3e896da348] Object does not exist on the server: [404] Object does not exist on the server
[8ab366a3428c0478474634ccbc096dea1eabdbcdc9566bd4d1dac045815b37e6] Object does not exist on the server: [404] Object does not exist on the server
[28d9adff2995decfc627cce04af69cfa7836b4837ec77fa74b29e7eef1aca731] Object does not exist on the server: [404] Object does not exist on the server
[816402c43c185df509c3e7914aefa49253eaa54ab44a4c0dd2a0dc25821fc65c] Object does not exist on the server: [404] Object does not exist on the server
[975e6821e0d244a9c9351ee8f150188f26febd1f1efa4017a4c5cf0ce0c9f451] Object does not exist on the server: [404] Object does not exist on the server
[f7b0a7c5cf4869376d03d8c67ab60bbe3b96627826de9dd4ed7d6d0e5c915f98] Object does not exist on the server: [404] Object does not exist on the server
[f2e8fea328c92335db5e9e2231a555a5172a965c56ab767981c4c7b0e2d6ce55] Object does not exist on the server: [404] Object does not exist on the server
[40d22776d88adf97a34a4fb7bfdac12ed7780f675a205124bbafb0787644028e] Object does not exist on the server: [404] Object does not exist on the server
[2599ba8b542b565a279f43f690ebc2b67215efc10938033d048aad196782a895] Object does not exist on the server: [404] Object does not exist on the server
[cd48b4ae6d4e1e44af5a887ac074fc0429d621f3c356c13e2e0523b3bf74e9ef] Object does not exist on the server: [404] Object does not exist on the server
[7247d715938655e83b91fda08143bf94a1e4cdf66b3852e1ca9917d9f324caad] Object does not exist on the server: [404] Object does not exist on the server
[2f8185d93bb7cbcce2630a056e35807f5a7a80ac9628570b77a253a068533b6f] Object does not exist on the server: [404] Object does not exist on the server
[35d6bb70fc46fb3ebb7172269f08d53991cdd2eaf587678c3c86fb0da591eb8c] Object does not exist on the server: [404] Object does not exist on the server
[9459b2940efe9bff1837c076f099e54793464bff59cefec98c2d7bc528453a30] Object does not exist on the server: [404] Object does not exist on the server
[09c17eb22270690248a0de721ee0fc2884fdd5df6254ec8f091eaef221f23e3e] Object does not exist on the server: [404] Object does not exist on the server
[408c82286afbe5fa397ac92071fcbc38145b324a2299ddd208617bdee2d21fa3] Object does not exist on the server: [404] Object does not exist on the server
[462933a23618f22198533951913649ed51807c1dcd9444b31680af6c10acf14e] Object does not exist on the server: [404] Object does not exist on the server
[1969a5dd1f10be215086788d2813f7a18cb05a4763235a96453818800562af3b] Object does not exist on the server: [404] Object does not exist on the server
[2acffacc07971331993da56ac1bbd26c31708726ac16574c63ad85ffbcc820d5] Object does not exist on the server: [404] Object does not exist on the server
[d955148600d1f206bca43b2459be7e6bc1736ca4abd3594b26a395bbe2cb87a6] Object does not exist on the server: [404] Object does not exist on the server
[c06336c73520cdc3336bdd113fd19f501453cfd193596414229b1f6ea65b56cb] Object does not exist on the server: [404] Object does not exist on the server
[ffcec4f6f0c804bfde97a83b700099361709bbf0e6cc9706b1db9afb25c6f546] Object does not exist on the server: [404] Object does not exist on the server
[78f020cb89ee5f04e3ef36ccd602b7994aa1dd2d6441bccf308fe02736e76303] Object does not exist on the server: [404] Object does not exist on the server
[d1c0db5b4d0efc5bf4e4f939784d992032c1d267bad4251ff3c244be6f784805] Object does not exist on the server: [404] Object does not exist on the server
[010db47441da01f70c7d221ccb6346040351319147f91a482dc2a4676fa39f37] Object does not exist on the server: [404] Object does not exist on the server
[13af71a1c6cd9f5ea321deecf4a5f49104093d90b16706e62dbc63cdf7f05749] Object does not exist on the server: [404] Object does not exist on the server
Failed to fetch some objects from 'https://github.com/TESTgroup-BNL/Global_Vcmax.git/info/lfs'

I think we need to be more careful of how we are using LFS and not blanket set certain files types to LFS unless they are larger than 2GBs and need to be set....I think this could cause us problems long term

[serbinsh]

@JulienLamour I recall now that you asked me to add LFS https://github.com/TESTgroup-BNL/Global_Vcmax/commit/e357dded222e6f2bb70f78ada5ef2e2bcc2f6af2

However it looks like perhaps at some point you didnt properly update the links of the LFS files, e.g. https://stackoverflow.com/questions/62852772/error-when-getting-git-lfs-files-object-does-not-exist-on-the-server-404-obj

so now I cant download the files just view them online.

We should probably remove blanket LFS and instead only track specific files

Presently I am unable to pull down the code and run it because certain files arent syncing correctly because of the LFS errors.

Do you still have the repo and all files on one of your machines? You may need to send me that so I can fix this and then untrack the LFS until we need it. Otherwise only you can see/access all of the files unless you fix the links by doing something like

git lfs push --all

[serbinsh]

Downloading Datasets/Albert_et_al_2018/Copy of Wu et al. 2019 spectra brazil.csv (1.7 MB)
Error downloading object: Datasets/Albert_et_al_2018/Copy of Wu et al. 2019 spectra brazil.csv (d791ec1): Smudge error: Error downloading Datasets/Albert_et_al_2018/Copy of Wu et al. 2019 spectra brazil.csv (d791ec1a83017c30201124615a77c08a87fa07e254dc2d7ea64e34e9ede5a5cb): [d791ec1a83017c30201124615a77c08a87fa07e254dc2d7ea64e34e9ede5a5cb] Object does not exist on the server: [404] Object does not exist on the server

Errors logged to /Users/sserbin/Data/GitHub/Global_Vcmax/.git/lfs/logs/20220919T122634.618469.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: Datasets/Albert_et_al_2018/Copy of Wu et al. 2019 spectra brazil.csv: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

exit status 128

[serbinsh]

@JulienLamour I just merged a PR that is attempting to fix LFS pointers but I wont really know if anything is missing. It will require that you check

Also do you remember what we needed LFS for? What specific files?

[serbinsh]

I have attempted to fix this by removing the LFS support and putting back older datasets. But this may be imperfect and you should perhaps double check i fixed everything properly