pip/conda installed HARK is missing some txt files

[llorracc]

I've just tried to run a couple of notebooks that use the cstwMPC tool and they failed because they couldn't find some input data ('USActuarial.txt' for example) that are supposed to be in the cstwMPC folder. This is very odd because:

1. Locally, they ARE in my (updated-to-the-latest-minute) local copy of HARK
2. On GitHub.com, they are in the cstwMPC directory
3. They have been there in every version of cstwMPC for years
4. I have done a pip install econ-ark --upgrade which did not retrieve them
5. They get created (and the notebooks work properly) when used by MyBinder
6. There's nothing in the .gitignore file that I can find that would be ignoring them

Maybe I should completely uninstall and reinstall HARK to see what I get...

[mnwhite]

It's possible that the cstwMPC code in HARK is pointing to the wrong directory for USactuarial.txt. In any case, I think we should delete cstwMPC from HARK and use the project repo instead.

On Mon, May 6, 2019 at 5:55 PM Christopher Llorracc Carroll < notifications@github.com> wrote:

I've just tried to run a couple of notebooks that use the cstwMPC tool and they failed because they couldn't find some input data ('USActuarial.txt' for example) that are supposed to be in the cstwMPC folder. This is very odd because:

1. Locally, they ARE in my (updated-to-the-latest-minute) local copy of HARK
2. On GitHub.com, they are in the cstwMPC directory https://github.com/econ-ark/HARK/tree/master/HARK/cstwMPC
3. They have been there in every version of cstwMPC for years
4. I have done a pip install econ-ark --upgrade which did not retrieve them
5. They get created (and the notebooks work properly) when used by MyBinder
6. There's nothing in the .gitignore file that I can find that would be ignoring them

Maybe I should completely uninstall and reinstall HARK to see what I get...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/279, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKRAFPMD3CZMGPPLY74PNDPUCSN3ANCNFSM4HLDWZBA .

[llorracc]

It can’t be pointing to the wrong directory because everything works when run with MyBinder — and when I run on my git-cloned HARK (as opposed to the system one).

Only think I can think of is that maybe at SOME point we had something in the .gitignore that caused these files to be ignored when I installed (I think by conda) on my machine.

But that still doesn’t explain why when I do a “pip install econ-ark —upgrade” it doesn’t repair the problem, since the files are not now being .gitignored

I’m guessing this would be fixed if I uninstalled and reinstalled HARK, but I’m posting as an issue in case the same thing happens to someone else.

On 2019-05-06, at 18:03, Matthew N. White notifications@github.com wrote:

It's possible that the cstwMPC code in HARK is pointing to the wrong directory for USactuarial.txt. In any case, I think we should delete cstwMPC from HARK and use the project repo instead.

On Mon, May 6, 2019 at 5:55 PM Christopher Llorracc Carroll < notifications@github.com> wrote:

I've just tried to run a couple of notebooks that use the cstwMPC tool and they failed because they couldn't find some input data ('USActuarial.txt' for example) that are supposed to be in the cstwMPC folder. This is very odd because:

1. Locally, they ARE in my (updated-to-the-latest-minute) local copy of HARK
2. On GitHub.com, they are in the cstwMPC directory https://github.com/econ-ark/HARK/tree/master/HARK/cstwMPC
3. They have been there in every version of cstwMPC for years
4. I have done a pip install econ-ark --upgrade which did not retrieve them
5. They get created (and the notebooks work properly) when used by MyBinder
6. There's nothing in the .gitignore file that I can find that would be ignoring them

Maybe I should completely uninstall and reinstall HARK to see what I get...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/econ-ark/HARK/issues/279, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKRAFPMD3CZMGPPLY74PNDPUCSN3ANCNFSM4HLDWZBA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

• Chris

---

Professor Christopher Carroll Dept of Economics Johns Hopkins University Mergenthaler 441 Baltimore, MD 21218-2685 (410)-516-7602 (o) (410)-516-7601 (Main office) mailto:ccarroll@jhu.edu http://econ.jhu.edu/people/ccarroll

Wisdom is only found in truth -- Chinese Fortune Cookie

The truth is rarely pure and never simple -- Oscar Wilde

[llorracc]

Same thing has happened to me again -- but now USactuarial.txt and similar files do not seem to be loading on MyBinder, so it is broken everywhere.

@shaunagm, let's add this to the agenda to fix tomorrow, probably by starting a new dev branch. You can reproduce the problem by running Macro-Implications-of-Very-Impatient-HHs.ipynb

It's weird because the files are clearly there on the GitHub version of HARK, and on my local copy (which I made the default by the pip -e trick), but somehow are not installed by pip install econ-ark.

[pkofod]

If you look at https://github.com/econ-ark/HARK/blob/6778cfd3f48b9706fe8d8aace7eaa4a04519f603/setup.py#L176 you will see that it refers to the module HARK.cstwMPC . It's possible that you can reference modules as "packages", but all examples I could find were like https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files That is, something like

    package_data={  # Optional
        'HARK': ['HARK/SolvingMicroDSOPs/SCFdata.csv',
             'HARK/cstwMPC/EducMortAdj.txt',
             'HARK/cstwMPC/SCFwealthDataReduced.txt',
             'HARK/cstwMPC/USactuarial.txt'
        ],
}

Again, I don't know setuptools well enough to know if the module reference should work. Also note, that the MANIFEST.in is not set up to allow this to work on Python 2 in any case https://github.com/econ-ark/HARK/blob/6778cfd3f48b9706fe8d8aace7eaa4a04519f603/MANIFEST.in

I tried installing HARK on a fresh linux install using pip and pip3, and got

so no text files!

[shaunagm]

The two methods of loading files were interfering with each other. It looks like, given the current setup of the manifest/setup files that we don't want most files included, since those lines are commented out of manifest.in. Perhaps this is because it makes the download process take longer? Anyway, please let me know if anyone does in fact want all the files included.

Fix was pretty simple given how long I spent on it . Remove the reference to the alternative method referencing Manifest.in in setup.py (the line include_package_data=True,) plus I also fixed the path for the SCFdata.csv file.

I've submitted a PR, although note that merging the PR won't immediately fix anything for you - because this impacts how packaging work, the changes won't have impact until the next release. I'll plan on doing a release by the end of the week, but if you want the fix before then you can get master from github, generate the distribution with python setup dist and re-install econ-ark from the local copy, pointing to the tar.gz file created by python setup dist, with something like pip install file:///ABSOLUTE-PATH-TO-FILES/HARK/dist/econ-ark-0.10.1.tar.gz.

[shaunagm]

Fixed by #332.