EmmaRocheteau / TPC-LoS-prediction

This repository contains the code used for Temporal Pointwise Convolutional Networks for Length of Stay Prediction in the Intensive Care Unit (https://dl.acm.org/doi/10.1145/3450439.3451860).
https://dl.acm.org/doi/10.1145/3450439.3451860
MIT License
74 stars 25 forks source link

Preproceesing eICU issue #10

Open Seham-Nasr opened 1 year ago

Seham-Nasr commented 1 year ago

I got the following error while running the pre-processing scripts by python3 -m eICU_preprocessing.run_all_preprocessing

File "/anaconda3/envs/TPC_Proj/lib/python3.8/runpy.py", line 194, in_run_module_as_main return _run_code(code, main_globals, None, File "/anaconda3/envs/TPC_Proj/lib/python3.8/runpy.py", line 87, in_run_code exec(code, run_globals) File "/eICU/TPC-LoS-prediction/eICU_preprocessing/run_all_preprocessing.py", line 18, in timeseries_main(eICU_path, test=False) File "/eICU/TPC-LoS-prediction/eICU_preprocessing/timeseries.py", line 228, in timeseries_main gen_timeseries_file(eICU_path, test) File "/eICU/TPC-LoS-prediction/eICU_preprocessing/timeseries.py", line 166, in gen_timeseries_file merged = timeseries_lab.loc[patient_chunk].append(timeseries_resp.loc[patien t_chunk], sort=False) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexing.py", line 967, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexing.py", line 1194, in _getitem_axis return self._getitem_iterable(key, axis=axis) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexing.py", line 1132, in _getitem_iterable keyarr, indexer = self._get_listlike_indexer(key, axis) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexing.py", line 1330, in _get_listlike_indexer keyarr, indexer = ax._get_indexer_strict(key, axis_name) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexes/multi.py", line 2587, in _get_indexer_strict self._raise_if_missing(key, indexer, axis_name) File "/anaconda3/envs/TPC_Proj/lib/python3.8/site-packages/pandas/co re/indexes/multi.py", line 2605, in _raise_if_missing raise KeyError(f"{keyarr[cmask]} not in index") KeyError: '[141939 142056 142476 142521 142560 146391 147447 149039 149606 15300 6\n 160529 162431 166572 166709 167391 167417 171174 175528 177651 178069\n 1788 58 179142 179554] not in index'

hanyin88 commented 1 year ago

Hey Seham - I ran into the same issues and it appears due to missing values in the lab csv... Have you find out a way to trouble shoot it? many thanks!

Update: fixed the issue after downgraded python form 3.9 to 3.6 and panda from higher version to the one specified by author.

Another update: turns still having issues with lower version of python/panda:

==> Processed 144000 patients... ==> Processed 148000 patients... Traceback (most recent call last): File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2657, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'patient'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/why217/Desktop/UIUC/cs598/TPC-LoS-prediction/eICU_preprocessing/run_all_preprocessing.py", line 18, in timeseries_main(eICU_path, test=False) File "/Users/why217/Desktop/UIUC/cs598/TPC-LoS-prediction/eICU_preprocessing/timeseries.py", line 229, in timeseries_main further_processing(eICU_path, test) File "/Users/why217/Desktop/UIUC/cs598/TPC-LoS-prediction/eICU_preprocessing/timeseries.py", line 205, in further_processing processed_timeseries.set_index('patient', inplace=True) File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/site-packages/pandas/core/frame.py", line 4178, in set_index level = frame[col]._values File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/site-packages/pandas/core/frame.py", line 2927, in getitem indexer = self.columns.get_loc(key) File "/Users/why217/opt/miniconda3/envs/TPS/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'patient'

Final update: ---finally figured out it's still the python/panda version issues with the new M1 MacBook. Everything resolved after I made a new env of python 3.6 and pip installed all requirements.

SpeedRanger commented 3 months ago

@hanyin88 I have downgraded my Python version but getting errors around installing torch=1.5.0. Can you share the final package versions that you have used to fix this issue please? Thanks.

SpeedRanger commented 3 months ago

Ok, these versions of packages worked for me(atleast for pre-processing):

requirements.txt:

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
absl-py=1.4.0=pypi_0
aiohttp=3.8.6=pypi_0
aiosignal=1.2.0=pypi_0
apscheduler=3.6.3=pypi_0
async-timeout=4.0.2=pypi_0
asynctest=0.13.0=pypi_0
attrs=22.2.0=pypi_0
backports-zoneinfo=0.2.1=pypi_0
cachetools=4.2.2=pypi_0
captum=0.2.0=pypi_0
certifi=2024.2.2=pypi_0
charset-normalizer=2.0.12=pypi_0
click=8.0.4=pypi_0
cloudpickle=3.0.0=pypi_0
colorama=0.4.6=pypi_0
colorlover=0.3.0=pypi_0
contourpy=1.1.1=pypi_0
cycler=0.10.0=pypi_0
dataclasses=0.8=pypi_0
decorator=4.4.2=pypi_0
dill=0.3.4=pypi_0
flask=2.0.3=pypi_0
fonttools=4.51.0=pypi_0
frozenlist=1.2.0=pypi_0
grpcio=1.48.2=pypi_0
idna=3.6=pypi_0
idna-ssl=1.1.0=pypi_0
imageio=2.15.0=pypi_0
importlib-metadata=7.1.0=pypi_0
importlib-resources=6.4.0=pypi_0
itsdangerous=2.0.1=pypi_0
jinja2=3.0.3=pypi_0
joblib=1.4.0=pypi_0
jsonpatch=1.32=pypi_0
jsonpointer=2.3=pypi_0
kiwisolver=1.4.5=pypi_0
llvmlite=0.36.0=pypi_0
markdown=3.3.7=pypi_0
markupsafe=2.0.1=pypi_0
matplotlib=3.2.2=pypi_0
multidict=5.2.0=pypi_0
multiprocess=0.70.12.2=pypi_0
networkx=2.5.1=pypi_0
numba=0.53.1=pypi_0
numpy=1.18.1=pypi_0
packaging=24.0=pypi_0
pandas=0.24.2=pypi_0
pathos=0.2.8=pypi_0
pillow=10.3.0=pypi_0
pip=21.3.1=pyhd8ed1ab_0
plotly=5.18.0=pypi_0
portalocker=2.7.0=pypi_0
pox=0.3.0=pypi_0
ppft=1.6.6.4=pypi_0
protobuf=3.19.6=pypi_0
pyparsing=3.1.2=pypi_0
python=3.6.15=h39d44d4_0_cpython
python-dateutil=2.9.0.post0=pypi_0
python-graphviz=0.19.1=pypi_0
python-telegram-bot=13.12=pypi_0
python_abi=3.6=2_cp36m
pytz=2024.1=pypi_0
pytz-deprecation-shim=0.1.0.post0=pypi_0
pywin32=305=pypi_0
requests=2.27.1=pypi_0
scikit-learn=0.20.2=pypi_0
scipy=1.4.1=pypi_0
seaborn=0.11.2=pypi_0
setuptools=58.0.4=py36ha15d459_2
shap=0.35.0=pypi_0
six=1.16.0=pypi_0
slackclient=2.9.4=pypi_0
slicer=0.0.7=pypi_0
tb-nightly=1.14.0a20190523=pypi_0
tenacity=8.2.2=pypi_0
threadpoolctl=3.4.0=pypi_0
torch=1.10.2=pypi_0
torchvision=0.11.3=pypi_0
tornado=6.1=pypi_0
tqdm=4.66.2=pypi_0
trixi=0.1.2.2=pypi_0
typing-extensions=4.11.0=pypi_0
tzdata=2024.1=pypi_0
tzlocal=4.2=pypi_0
ucrt=10.0.22621.0=h57928b3_0
umap-learn=0.4.6=pypi_0
urllib3=1.26.18=pypi_0
vc=14.3=hcf57466_18
vc14_runtime=14.38.33130=h82b7239_18
visdom=0.2.4=pypi_0
vs2015_runtime=14.38.33130=hcb4865c_18
websocket-client=1.3.1=pypi_0
werkzeug=2.0.3=pypi_0
wheel=0.37.1=pyhd8ed1ab_0
yarl=1.7.2=pypi_0
zipp=3.18.1=pypi_0