search

gentzkow / template_archive

19 stars 33 forks source link

Proposed revisions to standard template process #62

Closed snairdesai closed 1 year ago

snairdesai commented 2 years ago

This issue follows from #59, and our conversations with Hunt Allcott's RAs regarding their issues using the current template procedure across different OS. The proposed revisions in this issue will be distinct from the Docker development workflow proposals in #56, which we hope to add as a separate template workflow for capable users.

Below are a list of issues raised by the Hunt RAs:

Arjun (MacOS)

Zimei (WindowsOS)

In addition, a former RA has noted issues with running template in Mac M1 computers due to issues with Python versions in conda.

The deliverable for this issue will be a revised README for template which provides updated instructions for each of these conflicts, as well as some additional clarifications for users.

snairdesai commented 2 years ago

For @jc-cisneros and I (for now):

Here are a list of fixes which we anticipate will address each of the issues referenced above. We are testing these now.

  • [x] The chips.csv and tv.csv files in template were not properly copied down to the local repository.

This likely has to do with confusion around git-lfs usage. We are proposing that we explicitly note users should run git lfs pull to pull LFS objects. This should address the above.

  • [x] The run_all.py script did not fetch the correct directory due to an issue with the run_module() function in gslab_make.

This is an issue @jc-cisneros and I ran into as well, which has been newly addressed in commit bc55584 in PR #53 in gslab_econ.

  • [x] The instructions on command line usage for Lyx did not work.

@jc-cisneros and I believe the instructions on Command Line Usage for LyX could be updated to provide more clarity here. This will be included in our revised template proposals.

  • [x] There was an error in line 56 of make.py in the paper_slides module, where the Lyx document did not compile and was exported to PDF format.
  • [x] There was another error when attempting to compile the slides.lyx file with run_all.py.

We believe both of the above issues will be solved if the user installs an OS compatible TeX distributor. Instructions on how to do this will be included in the revised template.

  • [x] There were issues with Python version conflicts related to creating the conda environment.

This issue has been brought up frequently in the past. Zimei struggled with this in a different OS system: Windows. While this will be challenging to address in the standard template process, we do have two fixes in mind:

(i) Allowing users the option to install dependencies manually, rather than being forced to use conda. This has been addressed in Issue #18 in gslab-econ.

(ii) Suggesting the use of Docker containers to solve conflicts across distinct OS. We are working through this in #56.

  • [x] The directory in which check_setup.py was ran had to be changed in gslab_make.

This is likely a user issue relating to the directory in which command execution should be run. The check_setup.py file currently needs to be run within the /setup subfolder, rather in ~/root. We will clarify this with Zimei shortly.

  • [x] To append a new table in paper_slides, Zimei modified the tablefill() function to feature multiple tables.

We may open a new issue discussing this approach later, but this is not relevant for a template fix. It is rather an alternative (potentially useful) approach to making LyX edits.

  • [x] Zimei skipped the template instruction referencing: conda init (echo $0 | cut -d'-' -f 2).

This is likely a user issue, but raises an important flag which we will address in the revised template instructions. We have added instructions for MacOS/LinuxOS users suggesting they use bash terminals rather than zsh. We need to make a similar proposal for Windows users.

  • [x] A former RA has noted issues with running template in Mac M1 computers due to issues with Python versions in conda.

This is quite likely true across M1 users, and @jc-cisneros and I will be confirming this today. This will eventually be resolved if we were to integrate Docker into development and workflow, but we are thinking through a workaround for standard template as well.

snairdesai commented 2 years ago

@gentzkow Noting that we've opened this new issue (#62) to document and debug the above errors per our email exchange. We've closed the issue addressing conda dependencies (#59) and will integrate here. You can find the sample paragraph you requested to the AER data editor here.

gentzkow commented 2 years ago

Thanks @snairdesai. Email sent to data editor.

mthomas00 commented 2 years ago

Since you are revising the template, I thought I'd mention that it does not appear to run on Macs with M1 chips because python 3.7 is not available for these machines in conda. The template does appear to run with python 3.8, as set in conda_env.yaml. I have not thoroughly tested using python 3.8 approach, but python 3.9 produced errors. I'm still learning the new git-based template and gslab_make, but am so glad they are are available online. Thanks! Mike Thomas - former GSlab RA

snairdesai commented 2 years ago

Thank you @mthomas00! We'll add this to our item list.

gentzkow commented 2 years ago

@mthomas00 Good to hear from you!!! Thanks for the tip

meyer-carl commented 1 year ago

@mthomas00 @snairdesai Just as a heads up, we were discussing the same issue on https://github.com/gentzkow/CommitFlex/issues/114. I suggested a solution in the initial comment here that allows me to to stick with Python version 3.7 even on a new Mac with a M1 chip. The idea is to use Rosetta to have an environment that behaves as though it was on a Mac with an Intel processor.

gentzkow commented 1 year ago

Thanks @meyer-carl

Just to say, though, as I mentioned on the other thread I do not think we should be trying to stick to Python 3.7. Our goal should be to regularly update so we're using the most recent version of tools unless there's a specific reason not to. I do not think there is anything in our tools that should require python 3.7.

meyer-carl commented 1 year ago

@gentzkow yes, of course. I should have been clear that this is at best a stopgap solution.

snairdesai commented 1 year ago

@gentzkow:

Noting that @jc-cisneros and I are at a stage where we feel comfortable closing out this issue and initializing a new PR to merge back to master. The template revisions we propose in the PR will not be static, and some iteration will be required.

However, there are a few key updates detailed in our revisions which should probably live in master sooner rather than later to improve user efficiency (for example, setting a strict channel priority in conda reduced build time in CommitFlex from ~10 hours to ~10 minutes).

We can reopen this issue or iterate further if needed, once we have merged back to master. I've made a new development branch (issue62_template_revisions) where we will make proposed edits to the README.

snairdesai commented 1 year ago

Summary + Deliverable

In this issue (#62) following from #59, @jc-cisneros and I proposed a series of revision to the standard (i.e., non-Docker) template process. These edits will be dynamic, and more revisions will likely be required based on future user feedback.

The complete list of revisions (including edits to the template Wiki for "Command Line Usage") can be found in this Colab file. The relevant commit in our development branch is 9a4b206 below.

This issue continues in the associated PR (#70).