Not an issue more so a question/feature request

UTexas80 commented 3 months ago

Report an Issue / Request a Feature

I'm submitting a (Check one with "x") :

[ ] bug report
[x] feature request

Issue Severity Classification -

(Check one with "x") :

[ ] 1 - Severe
[x] 2 - Moderate
[ ] 3 - Low

Expected Behavior

It would be great if you could add python code within the munge folder and have it process as in the current comparable manner. I have added a "reticulate::source_python(python.py)" statement within my R code to initiate my python program to replicate the inherent Project Template functionality. Thank you for your time and consideration. ##### Current Behavior

Steps to Reproduce Behavior

insert short code snippets here

Screenshots

Version Information

Possible Solution

KentonWhite commented 3 months ago

I don't see why it couldn't be possible. I think it just requires a check for python files in load,.project.R:

  for (preprocessing.script in sort(dir(dir_name, pattern = munge_files())))
  {
    message(' Running preprocessing script: ', preprocessing.script)
    source(file.path(dir_name, preprocessing.script), local = .TargetEnv)
  }
  return(my.project.info)
}

just before the source (since it would need to run python).

Also need to change the munge file filter just above in munge_files()

If you would like to take a stab at adding this I'm super happy to help!

UTexas80 commented 3 months ago

Hello Kenton - Thank you for getting back to me so quickly, much appreciated. I would love to take a stab at adding this. I am looking through the load.project.R code and am excited to get this to run. Please let me know.

KentonWhite commented 3 months ago

The first step is to fork the project and see if you can modify load,.project.R to read and execute your python files. Once that is running we can work together on writing a test for the feature!

UTexas80 commented 3 months ago

Great Kenton! Thank you. Fork me, forgot all about it the forking. lol.

Onward.

On Tue, Mar 26, 2024 at 10:58 AM Kenton White @.***> wrote:

The first step is to fork the project and see if you can modify load,.project.R to read and execute your python files. Once that is running we can work together on writing a test for the feature!

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2020666529, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEL5VF2GRLSALG3QLALY2GEK5AVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRQGY3DMNJSHE . You are receiving this because you authored the thread.Message ID: @.***>

UTexas80 commented 3 months ago

Hello Kenton,

I forked ProjectTemplate and updated the munge_files function regex to include python files:

munge.files <- '[.][rR]|[.][pP][yY]$' # Add .py files

I was wondering how I would test this since the load.project.R code:

library("ProjectTemplate"); load.project()

points to the main branch?

KentonWhite commented 3 months ago

You van use devtools to install a local package:

https://devtools.r-lib.org/reference/install.html

UTexas80 commented 3 months ago

Very cool. Thank you so much. This is my first venture in updating an R library.

"Teach someone to fish..."

On Wed, Mar 27, 2024 at 9:55 AM Kenton White @.***> wrote:

You van use devtools to install a local package:

https://devtools.r-lib.org/reference/install.html

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2022833681, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEOD3VQCOH2AGEUBJVDY2LFUBAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRSHAZTGNRYGE . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite commented 3 months ago

No worries :) After the first one it gets kind of addicting!

UTexas80 commented 3 months ago

Got it to recognize the .py file in the munge folder.

Thank you for your help Kenton, much appreciated!

And you are right, this can be kind of addicting.

KentonWhite commented 3 months ago

Yay! Next steps is writing a test. I'm away for Easter weekend and can give some suggestions on the test to write when I'm back.

KentonWhite commented 3 months ago

I'm back from Easter Holidays.

For a package to be released to CRAN it needs to have tests for the features using testthat. In the tests folder there is a file called test-munge.R with the unit tests for the load function.

I think we should add a test to the final section called 'pass munge files to run' where instead of making a .R files we make a .py file and test that it is loaded in with load.project. All the test file needs to me is something that creates a variable that can be checked it it exists.

Could you take a stab at adding this test?

UTexas80 commented 3 months ago

Welcome back Kenton. Absolutely! I'd love to take a stab at this.

Glen

On Mon, Apr 1, 2024 at 2:49 PM Kenton White @.***> wrote:

I'm back from Easter Holidays.

For a package to be released to CRAN it needs to have tests for the features using testthat. In the tests folder there is a file called test-munge.R with the unit tests for the load function.

I think we should add a test to the final section called 'pass munge files to run' where instead of making a .R files we make a .py file and test that it is loaded in with load.project. All the test file needs to me is something that creates a variable that can be checked it it exists.

Could you take a stab at adding this test?

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2030345801, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GENOZBMOXL5ZN5TYVTDY3GTZTAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGM2DKOBQGE . You are receiving this because you authored the thread.Message ID: @.***>

UTexas80 commented 3 months ago

Hello Kenton,

I do not see a tet-munge.R file in the tests folder in the ProjectTemplate repository. Am I missing something?

Thank you. I look forward to hearing from you.

Glen

On Mon, Apr 1, 2024 at 2:49 PM Kenton White @.***> wrote:

I'm back from Easter Holidays.

For a package to be released to CRAN it needs to have tests for the features using testthat. In the tests folder there is a file called test-munge.R with the unit tests for the load function.

I think we should add a test to the final section called 'pass munge files to run' where instead of making a .R files we make a .py file and test that it is loaded in with load.project. All the test file needs to me is something that creates a variable that can be checked it it exists.

Could you take a stab at adding this test?

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2030345801, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GENOZBMOXL5ZN5TYVTDY3GTZTAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGM2DKOBQGE . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite commented 3 months ago

My bad, it is test-load.R. The munge section as at the very end.

UTexas80 commented 3 months ago

Thinking...

UTexas80 commented 3 months ago

Testing...

UTexas80 commented 3 months ago

Test Case: Interleaved Python and R Code Execution with Reticulate

Purpose: This test case verifies the ability of RStudio to seamlessly execute Python code interspersed with R code in a sequential order. The test utilizes the reticulate package to facilitate communication between R and Python environments.

Scope:

Functionality:
- Loading and executing Python scripts within R Studio.
- Importing Python libraries within R using reticulate.
- Reading and writing files from Python code.
- Capturing results from Python code within R.
Limitations:
- Focuses on basic functionalities.
- Doesn't test complex Python functionalities (e.g., object-oriented programming).

Test Design:

1. Test Environment:

RStudio IDE
testthat package installed
reticulate package installed
Python environment accessible from R

2. Test Data:

A temporary project directory will be created.
Python scripts will be dynamically generated within this directory.
An R script will be used to trigger the Python code execution.

3. Test Steps:

Create a temporary project directory.
Create a subdirectory named "munge" to store Python and R scripts.
Define two python test scripts:
- 01-test_data.py: Imports pandas and os, creates a dataFrame data, writes it to a CSV file (test_data_py.csv) and performs a calculation (e.g., sum of a column) and prints the result.
- 02-test_data.py: Imports pandas, os and sys, reads/writes the CSV file test_data_py.csv) created in 01-test_data.py, creates a dataframe py_data, defines a variable subdirectory, checks if the subdirectory variable exists in the python environment, passes the result to a variable data, prints whether y or n, writes a dynamically named dataframe either y.csv or n.csv to the munge subdirectory
Write the scripts to their respective files within the "munge" directory.
Use reticulate's source function to sequentially load the Python scripts from the R Project Template package.
Verify if the CSV files created by the Python script exist using except_false and file.exists from the testthat package.
Verify if the python variables exist in the R environment using except_false from the testthat package.
Execute the python script (01-test_data.py) to test capturing of the Python calculation result.
Execute the R script (01-test_data.R) to test capturing of the R result tibble.
Execute the python script (02-test_data.py) to test capturing of the Python environment result.
Execute the R script (02-test_data.R) to test capturing of the R result tibble.

4. Expected Results:

All interspersed Python and R scripts should be alphanumerically loaded successfully without errors.
The Python script (01-test_data.py) should capture the expected result from the Python calculation and the expect_true, file.exists assertion should pass.
The Python script (02-test_data.py) should capture the expected result from the Python environment and the expect_true, file.exists assertion should pass.
The R script (01-test_data.R) should capture the expected result from the R calculation and the expect_true, file.exists assertion should pass.
The R script (02-test_data.R) should capture the expected result from the R environment and the expect_true, file.exists assertion should pass.
The CSV files (test_data_py.csv, write_test_data_py.csv and y.csv) created by the Python script should exist and the expect_true, file.exists assertion should pass.
The CSV file (n.csv) should not be created by the Python script and the expect_false, file.exists assertion should pass.
The data file (data) created in the (01-test_data.py) script should not be written to the R Environment and the expect_false, assertion should pass.

5. Pass/Fail Criteria:

The test case passes if all expected results are met.
The test case fails if any errors occur during Python and R script execution, file operations, or if the R assertion fails.

Additional Considerations:

This test case can be further expanded to include more complex Python functionalities and error handling scenarios.
The test script content (e.g., library imports, data manipulation) can be customized based on specific use cases.
Ensure proper library installations and environment configurations for Python and R.

Conclusion: This test case demonstrates the basic functionality of running Python code interspersed with R code using reticulate. By successfully passing this test, we gain confidence in RStudio's ability to integrate Python code within the R environment, allowing for flexible data analysis workflows that leverage the strengths of both languages.

UTexas80 commented 3 months ago

Test Case: Interleaved Python and R Code Execution with Reticulate

Purpose: This test case verifies the ability of RStudio to seamlessly execute Python code interspersed with R code in a sequential order. The test utilizes the reticulate package to facilitate communication between R and Python environments.

Scope:

Functionality:
- Loading and executing Python scripts within R Studio.
- Importing Python libraries within R using reticulate.
- Reading and writing files from Python code.
- Capturing results from Python code within R.
Limitations:
- Focuses on basic functionalities.
- Doesn't test complex Python functionalities (e.g., object-oriented programming).

Test Design:

1. Test Environment:

RStudio IDE
testthat package installed
reticulate package installed
Python environment accessible from R

2. Test Data:

A temporary project directory will be created.
Python scripts will be dynamically generated within this directory.
An R script will be used to trigger the Python code execution.

3. Test Steps:

Create a temporary project directory.
Create a subdirectory named "munge" to store Python and R scripts.
Define two python test scripts:
- 01-test_data.py: Imports pandas and os, creates a dataFrame data, writes it to a CSV file (test_data_py.csv) and performs a calculation (e.g., sum of a column) and prints the result.
- 02-test_data.py: Imports pandas, os and sys, reads/writes the CSV file test_data_py.csv) created in 01-test_data.py, creates a dataframe py_data, defines a variable subdirectory, checks if the subdirectory variable exists in the python environment, passes the result to a variable data, prints whether y or n, writes a dynamically named dataframe either y.csv or n.csv to the munge subdirectory
Write the scripts to their respective files within the "munge" directory.
Use reticulate's source function to sequentially load the Python scripts from the R Project Template package.
Verify if the CSV files created by the Python script exist using except_false and file.exists from the testthat package.
Verify if the python variables exist in the R environment using except_false from the testthat package.
Execute the python script (01-test_data.py) to test capturing of the Python calculation result.
Execute the R script (01-test_data.R) to test capturing of the R result tibble.
Execute the python script (02-test_data.py) to test capturing of the Python environment result.
Execute the R script (02-test_data.R) to test capturing of the R result tibble.

4. Expected Results:

All interspersed Python and R scripts should be alphanumerically loaded successfully without errors.
The Python script (01-test_data.py) should capture the expected result from the Python calculation and the expect_true, file.exists assertion should pass.
The Python script (02-test_data.py) should capture the expected result from the Python environment and the expect_true, file.exists assertion should pass.
The R script (01-test_data.R) should capture the expected result from the R calculation and the expect_true, file.exists assertion should pass.
The R script (02-test_data.R) should capture the expected result from the R environment and the expect_true, file.exists assertion should pass.
The CSV files (test_data_py.csv, write_test_data_py.csv and y.csv) created by the Python script should exist and the expect_true, file.exists assertion should pass.
The CSV file (n.csv) should not be created by the Python script and the expect_false, file.exists assertion should pass.
The data file (data) created in the (01-test_data.py) script should not be written to the R Environment and the expect_false, assertion should pass.

5. Pass/Fail Criteria:

The test case passes if all expected results are met.
The test case fails if any errors occur during Python and R script execution, file operations, or if the R assertion fails.

Additional Considerations:

This test case can be further expanded to include more complex Python functionalities and error handling scenarios.
The test script content (e.g., library imports, data manipulation) can be customized based on specific use cases.
Ensure proper library installations and environment configurations for Python and R.

Conclusion: This test case demonstrates the basic functionality of running Python code interspersed with R code using reticulate. By successfully passing this test, we gain confidence in RStudio's ability to integrate Python code within the R environment, allowing for flexible data analysis workflows that leverage the strengths of both languages.

UTexas80 commented 3 months ago

https://github.com/UTexas80/ProjectTemplate-fork

KentonWhite commented 2 months ago

Been running into troubles getting this PR working with Travis.ci The issue is that the standard R build for travis runs on Xenial, which has Python2.7 and minimal support for Python3 (only Python3.5 and no pip3 support). This causes a problem with Reticulate, which really requires Python3.6 or higher.

Meanwhile, the Bionic build, which has great Python support, lacks good R support. It has R 4.0 out of box, which doesn't support dynamic loading of packages. This causes problems with the Tibble package, which requires dynamic loading.

The solution I'm exploring at the moment is either 1) installing Python3.6 directly on Xenial or 2) installing R release directly on Bionic.

UTexas80 commented 2 months ago

Hello Kenton,

Happy Monday. Thank you for the update. I appreciate it.

I think this is truly a great revision for the wonderful ProjectTemplate package that further expands its capabilities. I know you will figure it out!

I look forward to the update and look forward to hearing from you.

Glen

On Mon, Apr 22, 2024 at 11:19 AM Kenton White @.***> wrote:

Been running into troubles getting this PR working with Travis.ci The issue is that the standard R build for travis runs on Xenial, which has Python2.7 and minimal support for Python3 (only Python3.5 and no pip3 support). This causes a problem with Reticulate, which really requires Python3.6 or higher.

Meanwhile, the Bionic build, which has great Python support, lacks good R support. It has R 4.0 out of box, which doesn't support dynamic loading of packages. This causes problems with the Tibble package, which requires dynamic loading.

The solution I'm exploring at the moment is either 1) installing Python3.6 directly on Xenial or 2) installing R release directly on Bionic.

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2069855231, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEMXARXVEPY77W7KZXTY6US7PAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRZHA2TKMRTGE . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite commented 2 months ago

Travis build is working and everything is passing. Next step is pushing to CRAN.

UTexas80 commented 2 months ago

That’s fantastic Kenton. Exciting news! Thank you for letting me know.

I am looking forward to hearing about the results from CRAN.

Take care,

Glen

On Sat, Apr 27, 2024 at 6:44 PM Kenton White @.***> wrote:

Travis build is working and everything is passing. Next step is pushing to CRAN.

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2081220876, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEIDWAMQ6YRTRX6YPWDY7QS3PAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRGIZDAOBXGY . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite commented 1 month ago

The test servers on CRAN don't have pandas installed in their python installation. This is causing tests to fail and can't submit to CRAN with failing tests.

Is it possible to re-write the tests so that they don't use pandas or other packages not part of the base python installation?

UTexas80 commented 1 month ago

Good morning Kenton,

How are you? I have been wondering about the status of the ProjectTemplate modification.

I will rework the tests this weekend to just use the base python installation and let you know

Have a great Memorial Day,

Glen

On Mon, May 20, 2024 at 12:13 PM Kenton White @.***> wrote:

The test servers on CRAN don't have pandas installed in their python installation. This is causing tests to fail and can't submit to CRAN with failing tests.

Is it possible to re-write the tests so that they don't use pandas or other packages not part of the base python installation?

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2120767186, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEMEO35OH6LAZCFDVH3ZDIOKXAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQG43DOMJYGY . You are receiving this because you authored the thread.Message ID: @.***>

UTexas80 commented 1 month ago

Working on it...

UTexas80 commented 1 month ago

Test Complete: re-wrote the tests so that they don't use pandas or other packages not part of the base python installation.

KentonWhite commented 3 weeks ago

I'm not finding the new tests. Can you submit a new pull request with the changed tests please.

UTexas80 commented 3 weeks ago

Hello Kenton,

New pull request created.

If you have any questions, please let me know..

Thank you,

Glen

On Sun, Jun 23, 2024 at 3:59 PM Kenton White @.***> wrote:

I'm not finding the new tests. Can you submit a new pull request with the changed tests please.

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2185307470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEPPVPGNRBMSTB35WZ3ZI4SLTAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGMYDONBXGA . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite commented 2 weeks ago

On its way to CRAN!

UTexas80 commented 2 weeks ago

Awesome Kenton!!!!

Thank you for letting me know.

Have a great day,

Glen

On Mon, Jul 1, 2024 at 2:54 PM Kenton White @.***> wrote:

On its way to CRAN!

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2200784095, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEOUF4LFAMJNUXCTE63ZKGO2TAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQG44DIMBZGU . You are receiving this because you authored the thread.Message ID: @.***>

UTexas80 commented 6 days ago

Hello Kenton,

Just saw ProjectTemplate 0.11.0!!!!!!!

Thank you so much for letting me contribute!

Have a great week,

Glen

On Mon, Jul 1, 2024 at 3:26 PM Glen Falk @.***> wrote:

Awesome Kenton!!!!

Thank you for letting me know.

Have a great day,

Glen

On Mon, Jul 1, 2024 at 2:54 PM Kenton White @.***> wrote:

On its way to CRAN!

— Reply to this email directly, view it on GitHub https://github.com/KentonWhite/ProjectTemplate/issues/313#issuecomment-2200784095, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD23GEOUF4LFAMJNUXCTE63ZKGO2TAVCNFSM6AAAAABFEU2B66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQG44DIMBZGU . You are receiving this because you authored the thread.Message ID: @.***>

KentonWhite / ProjectTemplate