search

SEE-GEO / ccic

Chalmers Cloud Ice Climatology
MIT License
1 stars 2 forks source link

Generate training data #3

Closed simonpf closed 1 year ago

simonpf commented 2 years ago

Generate the training data for as many years as possible.

adriaat commented 1 year ago

The data generation has started today (2022-10-17) with the first CloudSat 2C-Ice file available and will proceed chronologically. The parameters used are the defaults from commit e7dc2f434ba9f043f5e73045e84d4d887f9134b3.

adriaat commented 1 year ago

In the first runs, the following happens:

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '574':
 'Dataset' object has no attribute 'iwp'

The tests also fail. I will look into with issue #9 this before continuing with the data generation.

simonpf commented 1 year ago

3ac01727eb4413253715c0934947cdc29011d7d1 fixes the remaining failing tests.

adriaat commented 1 year ago

Resumed after the fix in 021ec4d962faaea9a742eb5c5ec165976c23b039.

No errors so far but I'm only running with 2 processes in a 64GB-memory machine, and only providing the dates for which there are 2C-ICE R05 files available. More processes triggered errors, likely due to not enough memory. I am monitoring the memory usage, as well as see if I can use another 64GB-memory machine as well, all to try to generate the data faster.

adriaat commented 1 year ago

After running the process for a while, the authentication to get the 2C-ICE product fails. Below the error.

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/amell/ccic/ccic/bin/extract_training_data.py", line 116, in process_day
    granules = get_available_granules(date)
  File "/home/amell/ccic/ccic/data/cloudsat.py", line 421, in get_available_granules
    cls(filename) for filename in cls.get_available_files(date)
  File "/home/amell/ccic/ccic/data/cloudsat.py", line 171, in get_available_files
    files = cls.provider.get_files_by_day(date.year, day)
  File "/home/amell/pansat_ccic/pansat/pansat/download/providers/cloudsat_dpc.py", line 119, in get_files_by_day
    self.connection = SFTPConnection(
  File "/home/amell/pansat_ccic/pansat/pansat/download/providers/cloudsat_dpc.py", line 41, in __init__
    self._connect()
  File "/home/amell/pansat_ccic/pansat/pansat/download/providers/cloudsat_dpc.py", line 47, in _connect
    self.transport.connect(username=user_name, pkey=key)
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/site-packages/paramiko/transport.py", line 1388, in connect
    self.auth_publickey(username, pkey)
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/site-packages/paramiko/transport.py", line 1635, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/site-packages/paramiko/auth_handler.py", line 259, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/amell/miniconda3/envs/ccic/bin/ccic", line 33, in <module>
    sys.exit(load_entry_point('ccic', 'console_scripts', 'ccic')())
  File "/home/amell/ccic/ccic/bin/__init__.py", line 41, in ccic
    args.func(args)
  File "/home/amell/ccic/ccic/bin/extract_training_data.py", line 193, in run
    task.result()
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/home/amell/miniconda3/envs/ccic/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
paramiko.ssh_exception.AuthenticationException: Authentication failed.

In a secondary computer which should have a near identical setup, with the same ccic library version, the following error is raised many times. No more information is given. The XYZ is a number, like 1373

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule 'XYZ':
simonpf commented 1 year ago

Can you tell me for which dates these errors occur?

adriaat commented 1 year ago

Concerning the first error, that the authenication fails, it may be because the ssh key gets disabled. I cannot now manually establish an sftp connection into the server from the computer I get the authentication error. In the beginning the connection from both computers worked, but probably only for a while. Therefore, I believe this error may come from assigning my account to work with two different computers. I will take a step back now and run the process only in one computer for a while, and see if the same issue is raised.

Concerning the second error (The following error was encountered while processing CloudSat granule 'XYZ'), I need to investigate a bit about the dates, but to answer the question now, it was raised when running for July 2006. I give an update in a short while.

adriaat commented 1 year ago

To complement my previous comment, here's the stderr of the CloudSat granule error I got so far, which lists the granules. I want to make sure that I get the same errors with the other computer. Then I will know if the error comes from some environment issue or from the code/data repository.

stderr log 
ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '951':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '969':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '984':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '935':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '959':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '943':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '980':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '967':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '927':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '939':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '958':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '946':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '976':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '933':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '952':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '972':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '963':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '930':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '954':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '981':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '936':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '965':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '948':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '973':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '956':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '960':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '983':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '932':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '949':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '975':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '966':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '971':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '940':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '950':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '964':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '979':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '928':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '942':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '957':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '974':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '931':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '982':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '938':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '944':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '978':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '968':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '970':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '955':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '937':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '961':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '977':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '934':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '962':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '941':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '945':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '929':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1008':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '953':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '995':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1018':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '947':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1026':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '998':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1011':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1043':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1017':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1000':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '991':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1006':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1024':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1030':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '997':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1019':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1013':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1031':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '992':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1023':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1010':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1034':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1027':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1038':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1009':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '996':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1028':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1003':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1040':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '993':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1015':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1004':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '999':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1032':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1007':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1020':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '994':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1039':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1014':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1021':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1001':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1022':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1036':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1002':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1051':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1025':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1033':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1012':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1049':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1016':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1005':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1035':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1054':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1061':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1058':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1063':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1041':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1047':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1062':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1116':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1042':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1052':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1059':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1060':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1048':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1127':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1037':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1056':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1123':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1132':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1029':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1053':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1138':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1125':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1150':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1050':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1118':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1136':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1046':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1154':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1131':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1120':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1148':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1044':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1141':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1158':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1117':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1140':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1045':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1157':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1134':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1126':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1057':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1155':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1135':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1119':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1159':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1055':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1124':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1142':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1130':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1137':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1151':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1164':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1147':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1133':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1129':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1152':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1143':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1165':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1122':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1149':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1145':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1161':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1121':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1146':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1144':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1128':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1160':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1156':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1139':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1153':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1173':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1176':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1202':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1166':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1183':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1172':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1210':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1190':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1169':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1178':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1194':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1212':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1181':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1163':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1196':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1177':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1171':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1214':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1197':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1186':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1174':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1204':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1189':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1198':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1208':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1167':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1182':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1192':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1162':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1207':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1191':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1188':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1168':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1217':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1180':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1193':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1216':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1170':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1187':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1201':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1218':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1175':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1199':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1205':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1230':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1179':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1195':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1206':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1220':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1185':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1200':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1209':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1232':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1184':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1203':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1215':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1224':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1213':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1219':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1241':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1261':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1211':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1231':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1235':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1257':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1222':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1238':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1260':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1223':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1272':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1237':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1258':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1226':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1270':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1242':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1256':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1229':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1265':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1240':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1255':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1221':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1262':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1239':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1227':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1269':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1233':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1282':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1228':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1264':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1234':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1284':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1225':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1271':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1236':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1287':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1268':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1263':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1243':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1285':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1289':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1294':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1313':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1323':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1278':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1299':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1316':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1291':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1298':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1280':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1312':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1337':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1293':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1283':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1314':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1292':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1307':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1346':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1281':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1300':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1311':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1279':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1345':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1297':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1315':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1290':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1302':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1309':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1335':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1295':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1288':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1344':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1318':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1301':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1308':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1305':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1339':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1317':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1303':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1362':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1336':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1338':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1296':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1358':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1306':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1343':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1375':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1353':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1347':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1363':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1340':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1366':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1361':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1342':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1377':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1357':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1341':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1371':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1360':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1378':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1356':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1370':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1359':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1376':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1372':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1354':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1369':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1355':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1367':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1374':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1365':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1364':

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '1373':
adriaat commented 1 year ago

After running the process for about 2 hours, these two issues are not ocurring anymore*. So it seems that one user cannot have connections from more than one computer (more than one IP), and that I had some environment problem in the other machine where I tried to do a parallel processing, which raised the mysterious error without description.

To generate the data faster, we can look at using two accounts, or 'routing' the connection from the machine without an ssh key through the one with the working ssh key, so that only one account is used to download the data, but two machines to process it. I don't know how complicated this last option is.

*Another issue appeared, but only for one file so far (July 2007, granule 981):

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule '981':
 did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy', 'zarr']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html

I silently pass this (maybe the file was not downloaded correctly, it is corrupt, or something else happened), but if it occurs again/frequently it will have to be inspected.

simonpf commented 1 year ago

I suggest I start running 2008 on my machine and then we combine the data. It's anyways faster now than the old implementation so it shouldn't be to difficult to go through all the years until the battery failure within a couple of weeks.

adriaat commented 1 year ago

It sounds good. I will start to run only 2006 and 2007 in my machines. You might find the graph below relevant, which tells how much memory is consumed in one machine that has a small parallel process running. So I think n_workers=2 is the best option to keep things not collapsing.

memory_usage

I wrote the small script below so that it goes month by month silently, and logs any error while keeping the files and logs organized. Running it from the environment where ccic is installed, with the environment variables PANSAT_PASSWORD and CCIC_PATH. The latter, in my case, is /mnt/ccic.

import datetime
import os
import subprocess

# Process month per month
date_start_list = [
    datetime.datetime(2006,  8, 1), 
    datetime.datetime(2006,  9, 1), 
    datetime.datetime(2006, 10, 1), 
    datetime.datetime(2006, 11, 1), 
    datetime.datetime(2006, 12, 1), 
    datetime.datetime(2007, 1, 1), 
]

date_start = date_start_list[0]
for date_stop in date_start_list[1:]:

    month_process = date_start.strftime('%Y_%m')
    destination_folder = os.path.join(os.environ['CCIC_PATH'], "dataset/{:}".format(month_process))
    os.makedirs(destination_folder, exist_ok=True)

    ndays = (date_stop - date_start).days
    days_in_month = [(date_start + datetime.timedelta(days=d)).strftime('%-d') for d in range(ndays)]
    dates_string = ' '.join(days_in_month)

    destination_stderr = os.path.join(os.environ['CCIC_PATH'], "logs/{:}.stderr".format(month_process))
    destination_stdout = os.path.join(os.environ['CCIC_PATH'], "logs/{:}.stdout".format(month_process))
    with open(destination_stderr, 'a') as stderr_handler, open(destination_stdout, 'a') as stdout_handler:
        subprocess.run("ccic extract_training_data {:} {:} {:} {:} --n_workers 2".format(date_start.year, date_start.month, dates_string, destination_folder).split(),
            stdout=stdout_handler, stderr=stderr_handler)

    # Iterate to next date
    date_start = date_stop
adriaat commented 1 year ago

FYI, I still got the mysterious error

ccic.bin.extract_training_data (ERROR     ) :: The following error was encountered while processing CloudSat granule 'XYZ':

in the computer where I mount the directory from the other computer. The solution was to download to a local directory, update the variable CCIC_PATH to /data/ccic_local/, and I will copy the data to the remote directory afterwards.

adriaat commented 1 year ago

Part of 2006, 2007, and 2008 years' data downloaded.

To do:

~Currently (1 November 2022) this is paused due to issues with the CloudSat DPC SFTP server.~

The download is resumed but at a slower pace, now because of problems with the local computers.

simonpf commented 1 year ago

One issue with the training data seems to be that cloud_mask is always 1. For the current training, I infer the value of cloud_mask from cloud_class on the fly, so it's not that critical but I guess we should, at some point, figure out where that went wrong.

adriaat commented 1 year ago

The problem is likely to reside in pansat, here. The callback function should be _cloud_scenario_to_cloud_scenario_flag and not _cloud_scenario_to_cloud_class.

The consequence of this typo is that whenever there are no clouds, but that was successfully determined by the algorithm, we are assigning -1 (invalid value) to the cloud class labels (lines of code).

I tested this correction with the file 2010017014344_19801_CS_2B-CLDCLASS_GRANULE_P1_R05_E03_F00.hdf, and the number of "valid" profiles changes from 19666 to all of them (37081).

If we need to correct this, then we should discuss this: I do not think the code does what was intended. The array cloud_mask is of type bool, so assigning -1 to where there are non-valid cells sets them to True, that is, as "cloud present".

adriaat commented 1 year ago

I ran a small test to assess the impact of this

One issue with the training data seems to be that cloud_mask is always 1. For the current training, I infer the value of cloud_mask from cloud_class on the fly, so it's not that critical but I guess we should, at some point, figure out where that went wrong.

by correcting the callback used in pansat, downloading all 2B-CLDCLASS data for 2007, and counting the number of profiles (rays) for which the cloud scenario could not be determined, as well as the number of bins in each ray for which the scenario could not be determined. The code is here, and the results are:

Therefore, and extrapolating from the 2007 data, this error turns out to be relatively small. They also seem to be rather isolated, as these are the ray indices which presented problems (they are sorted chronologically, starting from zero):

10107759,  10107760,  14108195,  14108196,  59918999,  59919000, 120769724, 120769725, 162849885, 162849886

(The test took 25 minutes to complete)

Albeit, I believe, we did not save from which ray each data point is coming from, we could correct this a posteriori, by manually identifying if any of the problematic rays was used, and if yes, finding the values used.

adriaat commented 1 year ago

The generation of training data for the period July 2006 – March 2011 has been completed, subject to the known issues to date. The data is available in the shared computer.

The data consists of 510_678 files (45.35 GB), containing about 1.4E8 (1.7E7) collocated profiles with GPM MERGEIR (GridSat).

Less than 10 granules could not be processed, and a few more (40) errors were raised when processing some granules. I believe these errors may be related to the CloudSat files themselves. More information can be found in the folder logs in the shared computer, organized by months.

The year 2008 was also re-generated, but only resulted in an increase of 0.5% of the previously generated files for 2008, which indicates the download went pretty well for that year.

If we generate new or more training data, for this given period or for the period that contains daytime observations, we should first fix #12. I also suggest to add an attribute which indicates the CloudSat granule used for the collocations, and an option to download and process specific granules (I did this last thing to solve some of the problems when correcting problems raised when processing for some granules, but unfortunately I did not save nor commit the code). Finally, the variables should include their units.

adriaat commented 1 year ago

After the exploration of the training data in #4, here is a list of issues to analyse or fix with the training data for posterior data extraction:

adriaat commented 1 year ago

A new training dataset was extracted with #15, which addresses all issues, by minimizing the issue with collocations towards the antimeridian, higher souther latitudes and 0º and 5º N, and resolves all others.