search

COSIMA / master_index

Scripts to generate COSIMA master index
0 stars 0 forks source link

Multiple copies of 01deg_jra55v140_iaf in database #8

Closed aidanheerdegen closed 3 years ago

aidanheerdegen commented 3 years ago

This data is in two locations and indexed twice in the database.

$ ls /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf                                     
metadata.yaml  output020  output041  output062  output083  output104  output125  output146  output167  output188  output209  output230
output000      output021  output042  output063  output084  output105  output126  output147  output168  output189  output210  output231
output001      output022  output043  output064  output085  output106  output127  output148  output169  output190  output211  output232
output002      output023  output044  output065  output086  output107  output128  output149  output170  output191  output212  output233
output003      output024  output045  output066  output087  output108  output129  output150  output171  output192  output213  output234
output004      output025  output046  output067  output088  output109  output130  output151  output172  output193  output214  output235
output005      output026  output047  output068  output089  output110  output131  output152  output173  output194  output215  output236
output006      output027  output048  output069  output090  output111  output132  output153  output174  output195  output216  output237
output007      output028  output049  output070  output091  output112  output133  output154  output175  output196  output217  output238
output008      output029  output050  output071  output092  output113  output134  output155  output176  output197  output218  output239
output009      output030  output051  output072  output093  output114  output135  output156  output177  output198  output219  output240
output010      output031  output052  output073  output094  output115  output136  output157  output178  output199  output220  output241
output011      output032  output053  output074  output095  output116  output137  output158  output179  output200  output221  output242
output012      output033  output054  output075  output096  output117  output138  output159  output180  output201  output222  output243
output013      output034  output055  output076  output097  output118  output139  output160  output181  output202  output223           
output014      output035  output056  output077  output098  output119  output140  output161  output182  output203  output224           
output015      output036  output057  output078  output099  output120  output141  output162  output183  output204  output225           
output016      output037  output058  output079  output100  output121  output142  output163  output184  output205  output226           
output017      output038  output059  output080  output101  output122  output143  output164  output185  output206  output227           
output018      output039  output060  output081  output102  output123  output144  output165  output186  output207  output228           
output019      output040  output061  output082  output103  output124  output145  output166  output187  output208  output229           
$ ls /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf
error_logs     output042  output087  output132  output177  output222   restart091
git-runlog     output043  output088  output133  output178  output223   restart095
metadata.yaml  output044  output089  output134  output179  output224   restart099
output000      output045  output090  output135  output180  output225   restart103
output001      output046  output091  output136  output181  output226   restart107
output002      output047  output092  output137  output182  output227   restart111
output003      output048  output093  output138  output183  output228   restart115
output004      output049  output094  output139  output184  output229   restart119
output005      output050  output095  output140  output185  output230   restart123
output006      output051  output096  output141  output186  output231   restart127
output007      output052  output097  output142  output187  output232   restart131
output008      output053  output098  output143  output188  output233   restart135
output009      output054  output099  output144  output189  output234   restart139
output010      output055  output100  output145  output190  output235   restart143
output011      output056  output101  output146  output191  output236   restart147
output012      output057  output102  output147  output192  output237   restart151
output013      output058  output103  output148  output193  output238   restart155
output014      output059  output104  output149  output194  output239   restart159
output015      output060  output105  output150  output195  output240   restart163
output016      output061  output106  output151  output196  output241   restart167
output017      output062  output107  output152  output197  output242   restart171
output018      output063  output108  output153  output198  output243   restart175
output019      output064  output109  output154  output199  pbs_logs    restart179
output020      output065  output110  output155  output200  restart003  restart183
output021      output066  output111  output156  output201  restart007  restart187
output022      output067  output112  output157  output202  restart011  restart191
output023      output068  output113  output158  output203  restart015  restart195
output024      output069  output114  output159  output204  restart019  restart199
output025      output070  output115  output160  output205  restart023  restart203
output026      output071  output116  output161  output206  restart027  restart207
output027      output072  output117  output162  output207  restart031  restart211
output028      output073  output118  output163  output208  restart035  restart215
output029      output074  output119  output164  output209  restart039  restart219
output030      output075  output120  output165  output210  restart043  restart223
output031      output076  output121  output166  output211  restart047  restart227
output032      output077  output122  output167  output212  restart051  restart231
output033      output078  output123  output168  output213  restart055  restart235
output034      output079  output124  output169  output214  restart059  restart239
output035      output080  output125  output170  output215  restart063  restart243
output036      output081  output126  output171  output216  restart067  run_summary_home_156_aek156_payu_01deg_jra55v140_iaf.csv
output037      output082  output127  output172  output217  restart071
output038      output083  output128  output173  output218  restart075
output039      output084  output129  output174  output219  restart079
output040      output085  output130  output175  output220  restart083
output041      output086  output131  output176  output221  restart087
aidanheerdegen commented 3 years ago

01deg_jra55v140_iaf_cycle2 is similar, it appears in both places.

aekiss commented 3 years ago

/g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf and /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle2 are mostly just restarts and shouldn't be indexed. Perhaps they should be moved elsewhere?

IIRC /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf/output* is is just the remains left behind after I rsynced to /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf. Sorry I forgot to tidy this up.

aidanheerdegen commented 3 years ago

Perhaps they should be moved elsewhere?

Good idea. Maybe make a directory /g/data/ik11/restarts and make a directory hierarchy in there the same as in /g/data/ik11/outputs?

If you do this let me know when it's done and I re-run the job to update the DB. If you recall, it is best to leave the directory there so the files can all be deleted when the DB is updated, then the directory can be deleted. At least that is how it used to work ...

aekiss commented 3 years ago

OK I've moved /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf and /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle2 to /g/data/ik11/restarts/access-om2-01/