Update dependencies required in README to run all the tests.

manisnesan commented 2 years ago

PR to update README

sentencepiece is no more version constrained as per dev_requirements.
we don't want to spread out the dependencies in multiple places as it may likely be get out of date.

manisnesan commented 2 years ago

@muellerzr - I created this simple PR to understand first time contributor workflow.

muellerzr commented 2 years ago

I'd rather we state exactly how they should install it. E.g. pip install -e .[dev]

As those should be valid in the setup.py. if you need to change the requirements in settings.ini, don't hesitate to do that.

manisnesan commented 2 years ago

https://discord.com/channels/689892369998676007/933769297052975194/942610147614793728 - Note to self about Error in installing Dev.

Capturing the gist from discord

Ideally this should have been caught in CI.
CI may not have caught this ~~due to quiet option~~ because
- .github/workflows/main.yml test-notebooks runs using the container fastai/fastai . The image is built using docker-containers repository. This image set to use python 3.8. Hence CI only tests with python version 3.8.
- This is likely a risk for the project as the dependencies issues like this may fall off of our radar if we are testing only with specific python version 3.8 but setting.ini indicates min_python = 3.6. Ideally we may want to run test-notebooks on python versions (3.6, 3.7, 3.8, 3.9) that fastai support.
- Potential Solution to catch this early https://blog.deepjyoti30.dev/tests-github-python.
azuremlcallback is run with #all_slow nbdev incantation which isn't run on CI.

EDIT: Updated the reason why CI did not catch this dependency issue.

muellerzr commented 2 years ago

@manisnesan proposed solution I'd love to see:

Eliminate (for now) having azureml in the dev_requirements in settings.
State that when building the docs, ignore errors for azure integration.

manisnesan commented 2 years ago

@muellerzr I created a new conda environment and did pip install -e ".[dev]" for both python versions 3.6 & 3.8. Both were successfully able to install azureml-sdk along with transitive dependencies (azureml-train-automl-client -> this is the one you mentioned having trouble with). Could you provide the python version you are having issues with??

Python 3.6

(azureml_test_3_6) ➜  fastai-docment-sprint git:(manisnesan_test_dep) ✗ python --version       
Python 3.6.13 :: Anaconda, Inc.

$ pip freeze | grep azure                    
azure-common==1.1.28
azure-core==1.21.1
azure-graphrbac==0.61.1
azure-identity==1.7.0
azure-mgmt-authorization==0.61.0
azure-mgmt-containerregistry==8.2.0
azure-mgmt-core==1.3.0
azure-mgmt-keyvault==9.3.0
azure-mgmt-resource==20.1.0
azure-mgmt-storage==19.0.0
azureml-automl-core==1.38.0
azureml-core==1.38.0.post2
azureml-dataprep==2.26.0
azureml-dataprep-native==38.0.0
azureml-dataprep-rslex==2.2.0
azureml-dataset-runtime==1.38.0
azureml-pipeline==1.38.0
azureml-pipeline-core==1.38.0
azureml-pipeline-steps==1.38.0
azureml-sdk==1.38.0
azureml-telemetry==1.38.0
azureml-train-automl-client==1.38.0.post2
azureml-train-core==1.38.0
azureml-train-restclients-hyperdrive==1.38.0
msrestazure==0.6.4

Python 3.8

(azureml_test) ➜  fastai-docment-sprint git:(manisnesan_test_dep) ✗ python --version                                   
Python 3.8.12                                                                                                          
(azureml_test) ➜  fastai-docment-sprint git:(manisnesan_test_dep) ✗ pip freeze | grep azure                                                                                                                                                   
azure-common==1.1.28                                                                                                                                                                                                                          
azure-core==1.21.1                                                                                                                                                                                                                            
azure-graphrbac==0.61.1                                                                                                                                                                                                                       
azure-identity==1.7.0                                                                                                                                                                                                                         
azure-mgmt-authorization==0.61.0                                                                                       
azure-mgmt-containerregistry==8.2.0                                                                                                                                                                                                           
azure-mgmt-core==1.3.0                                                                                                                                                                                                                        
azure-mgmt-keyvault==9.3.0                                                                                                                                                                                                                    
azure-mgmt-resource==20.1.0                                                                                                                                                                                                                   
azure-mgmt-storage==19.0.0                                                                                                                                                                                                                    
azureml-automl-core==1.38.0                                                                                                                                                                                                                   
azureml-core==1.38.0.post2                                                                                                                                                                                                                    
azureml-dataprep==2.26.0                                                                                                                                                                                                                      
azureml-dataprep-native==38.0.0                                                                                                                                                                                                               
azureml-dataprep-rslex==2.2.0                                                                                                                                                                                                                 
azureml-dataset-runtime==1.38.0                                                                                                                                                                                                               
azureml-pipeline==1.38.0                                                                                                                                                                                                                      
azureml-pipeline-core==1.38.0                                                                                                                                                                                                                 
azureml-pipeline-steps==1.38.0                                                                                                                                                                                                                
azureml-sdk==1.38.0                                                                                                                                                                                                                           
azureml-telemetry==1.38.0                                                                                                                                                                                                                     
azureml-train-automl-client==1.38.0.post2                                                                                                                                                                                                     
azureml-train-core==1.38.0                                                                                                                                                                                                                    
azureml-train-restclients-hyperdrive==1.38.0                                                                                                                                                                                                  
msrestazure==0.6.4

manisnesan commented 2 years ago

The root cause is azureml-train-automl-client requires Python >=3.6, <3.9. Hence it may not work for python environments greater than 3.9.

Based on this, I will go with

Eliminate (for now) having azureml in the dev_requirements in settings.

State that when building the docs, ignore errors for azure integration.

manisnesan commented 2 years ago

@muellerzr - Completed the steps.

muellerzr commented 2 years ago

@manisnesan don't forget to also update index.ipynb, as that's what builds the README

manisnesan commented 2 years ago

@muellerzr - My bad. Now I know why there is a conflict.:) I will update index.original.ipynb with the changes.

manisnesan commented 2 years ago

Closing this PR. See https://github.com/muellerzr/fastai-docment-sprint/pull/39

fastaidocsprint / fastai

Update dependencies required in README to run all the tests. #2