Make caching optional to enable cloud computing and computing in highly restricted environments

[meteoDaniel]

The implemented caching tries to write data into the /home/ which is not possible in cloud computing and on machines with restricted write rights.

So there should be a flag to enable/disable caching.

{
  "errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1051'",
  "errorType": "OSError",
  "stackTrace": [
    "  File \"/usr/local/lib/python3.9/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n",
    "  File \"/usr/local/lib/python3.9/imp.py\", line 171, in load_source\n    module = _load(spec)\n",
    "  File \"<frozen importlib._bootstrap>\", line 711, in _load\n",
    "  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 790, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
    "  File \"/app/src/app.py\", line 4, in <module>\n    from src.pipelines import (\n",
    "  File \"/app/src/pipelines.py\", line 9, in <module>\n    from src.radar_data.german_radar_data_download import process_sweep_data_download\n",
    "  File \"/app/src/radar_data/german_radar_data_download.py\", line 8, in <module>\n    from wetterdienst.dwd.radar import (\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/__init__.py\", line 7, in <module>\n    from wetterdienst import dwd\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/dwd/__init__.py\", line 6, in <module>\n    from wetterdienst.dwd import forecasts, observations, radar\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/dwd/observations/__init__.py\", line 4, in <module>\n    from wetterdienst.dwd.observations.api import (\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/dwd/observations/api.py\", line 13, in <module>\n    from wetterdienst.dwd.index import _create_file_index_for_dwd_server\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/dwd/index.py\", line 14, in <module>\n    from wetterdienst.util.cache import (\n",
    "  File \"/usr/local/lib/python3.9/site-packages/wetterdienst/util/cache.py\", line 23, in <module>\n    os.makedirs(cache_dir)\n",
    "  File \"/usr/local/lib/python3.9/os.py\", line 215, in makedirs\n    makedirs(head, exist_ok=exist_ok)\n",
    "  File \"/usr/local/lib/python3.9/os.py\", line 215, in makedirs\n    makedirs(head, exist_ok=exist_ok)\n",
    "  File \"/usr/local/lib/python3.9/os.py\", line 225, in makedirs\n    mkdir(name, mode)\n"
  ]
}

A work around could be to use /tmp directory which is accesable in AWS Lambda e.g. Another solution could be to have the ability to define the caching directory. But I would prefer to set a flag.

Actually I am not sure if it is possible to switch on/off decorators. So the easiest way would be to set the directory path from outside.

[amotl]

Dear Daniel,

There should be a flag to enable/disable caching.

I support your suggestion to optionally disable caching at all.

Actually I am not sure if it is possible to switch on/off decorators.

That would not be so easy, right.

So the easiest way would be to set the directory path from outside.

I believe the easiest way would be to use the "memory" backend of dogpile.cache like we are currently doing on Windows. So, let's just introduce a WD_CACHE_DISABLE environment variable for that purpose, as suggested by @gutzbenj, see https://github.com/earthobservations/wetterdienst/pull/418#discussion_r617726706? Would you support that way by adjusting your patch #418 accordingly?

Let me know if your time is too scarce to follow up on that. We might support you on giving this patch the finishing touch if you agree on the suggestions.

With kind regards, Andreas.