File paths only work on Yannick's laptop

[nickmalleson]

File paths point to a specific location on Yannick's laptop. E.g os.chdir("C:/Users/earyo/Dropbox/Arbeit/postdoc_leeds/ABM_python_first_steps/implement_own_covid_policy_model"). This means that other people wont be able to run the code. You should use relative paths.

In a notebook this is quite straightforward as I think the notebook directory is always set to be the place where the notebook is stored. E.g. if I open run_base_model_and_filter_with_plotting_jupyter.ipynb and do %pwd (it's a notebook command that says show me the _p_resent _w_orking _d_irectory) then I get /Users/nick/gp/Covid_policy_response_abm, which is the directory on my machine where the notebook is stored. So any other files can be referenced from that directory.

This means that lines like this:

with open('C:/Users/earyo/Dropbox/Arbeit/postdoc_leeds/ABM_python_first_steps/implement_own_covid_policy_model/data/correlation_between_policies_and_metrics.csv') as f:

can be replaced with simply:

with open('./data/correlation_between_policies_and_metrics.csv') as f:

(the . is unnecessary but I like it because it says explicitly that the directory to start from is the current directory)

Sometimes it is slightly trickier with code and packages, but we can come on to that if it's a problem...

[yannickoswald]

oh ok that seems simple enough. i didnt find somehow a clear explanation how to do it. so that is why i asked keiran to help out with these things.

[nickmalleson]

Quick update on the use of file paths in code

I haven't found an instance where setting the directory is necessary yet. For example in https://github.com/eeyouol/Covid_policy_response_abm/blob/master/code/run_base_model_only_parallelized.py I don't think the following line is necessary so long as the python interpreter is started from the correct directoy:

https://github.com/eeyouol/Covid_policy_response_abm/blob/9f11772661c3169f8c6dfbef9729821b62e0e38f/code/run_base_model_only_parallelized.py#L48

E.g. to run the above script, you could start in the root project directory (Covid_policy_response_abm) and use:

python code/run_base_model_only_parallelized.py

(or, in an IDE, you can set the project working directory if necessary (although by default I reckon the IDE will set the directory to be the root project directory anyway)).

Then in that script the os.chdir is unnecessary because python will automatically 'be' in that directory.

Similarly for loading data you could replace

https://github.com/eeyouol/Covid_policy_response_abm/blob/9f11772661c3169f8c6dfbef9729821b62e0e38f/code/run_base_model_only_parallelized.py#L48

with

    with open('./data/agent_data_v2.csv') as f:
        agent_data = pd.read_csv(f, encoding='unicode_escape')

[nickmalleson]

And in case my explanation isn't clear, here's what ChatGTP says about it :-) (although I think the os.getcwd() is unnecessary...

[ksuchak1990]

@nickmalleson and @eeyouol - this is pretty much what I was looking at last night, but haven't had a chance to go through all of the files (we need to do something similar for the notebook in the repo root).

[yannickoswald]

since i want to submit this friday, i will try to implement this now. let me know if you have capacity to do this before friday @ksuchak1990

[yannickoswald]

I am still working on those paths now with . and .. etc. Seems more tricky than i thought because of complex dependencies.

[yannickoswald]

BIG UPDATE @ksuchak1990 @nickmalleson: I solved this now I believe.

I updated all the file paths to relative paths. the ones referring to the data need to be

    with open('../data/agent_data_v2.csv') as f:
        agent_data = pd.read_csv(f, encoding='unicode_escape')

with double dot .. because they are in the folder "code" and want to access from the parent directory the folder "data".

It all runs now if set my working directory in the anaconda prompt to

(base) PS C:\Users\earyo\Dropbox\Arbeit\postdoc_leeds\ABM_python_first_steps\implement_own_covid_policy_model\code>

but obviously should work also from other environments and other paths if the folder structure is as preserved in GitHub

Please let me know if you can confirm this?

[yannickoswald]

the relative paths continue to mess stuff up. I dont get it really to work with the notebook at all. before without notebook everything did well. sorry. have to see how this will work

[yannickoswald]

the relative paths continue to mess stuff up. I dont get it really to work with the notebook at all. before without notebook everything did well. sorry. have to see how this will work

i classify this a secondary issue now, because while the notebook is much nicer, the actual reproduction @ksuchak1990 also works just with the file "run_base_model_and_filter_with_plotting.py". Please test this. if this works the notebook is just extra

[ksuchak1990]

the relative paths continue to mess stuff up. I dont get it really to work with the notebook at all. before without notebook everything did well. sorry. have to see how this will work

i classify this a secondary issue now, because while the notebook is much nicer, the actual reproduction @ksuchak1990 also works just with the file "run_base_model_and_filter_with_plotting.py". Please test this. if this works the notebook is just extra

I'm working through this now. So far, it looks like it works but will update later today.

[yannickoswald]

the relative paths continue to mess stuff up. I dont get it really to work with the notebook at all. before without notebook everything did well. sorry. have to see how this will work

i classify this a secondary issue now, because while the notebook is much nicer, the actual reproduction @ksuchak1990 also works just with the file "run_base_model_and_filter_with_plotting.py". Please test this. if this works the notebook is just extra

I'm working through this now. So far, it looks like it works but will update later today.

I changed the notebook now again. It works (partially). Only if I set the notebook path explicitly with the os package. I described this in the first comment in the notebook.

import os
##### IMPORTANT ######
#### CHANGE YOUR WORKING DIRECTORY Explicitly YOURSELF to "mypath/covpol" 
#### (whereever you keep covpol)
#### e.g.os.chdir("folder1/folder2/covpol")
#os.chdir("./covpol") THIS works only if Jupyter knows your working from a specific folder
os.chdir("C:/Users/earyo/Dropbox/Arbeit/postdoc_leeds/ABM_python_first_steps/implement_own_covid_policy_model/covpol")

[nickmalleson]

I made some minor changes in the most recent pull request but other than that I can confirm most works well for me

https://github.com/yloswald/covpol/pull/25

I've not tried to run the more time intensive scripts, but all that I did run worked fine :-)