Doesn't seem to be interpolating values from prompts

acedrew commented 1 year ago

None of the templating tags are rendered in the generated agent

craig8 commented 1 year ago

@acedrew what version of ubuntu are you working on with what version of python? Have you moved to 3.10 or 11? Thanks @kefeimo please look into this for me

kefeimo commented 1 year ago

@acedrew , would you mind listing the command(s) you ran, and pasting the (unexpected) prompt it shows (either text or screenshots). I will have to try to reproduce the issue first. (Other information might be needed for further investigation.) Thank you.

acedrew commented 1 year ago

Ubuntu 20.04 Python 3.8.10

Prompts look fine, just doesn't use any of the input

copier "https://github.com/VOLTTRON/copier-poetry-volttron-agent.git" ./EmptyFolder

kefeimo commented 1 year ago

Demo bug-reporting template.

This is an example to demonstrate a bug-reporting workflow that helps package developers/issue handlers to reproduce the issue.

Environment setup

Ubuntu 20.04 VM
Python 3.8.10

Create a virtual environment

(Assuming the a sandbox testing path is at "~/project-local") $ cd ~/project-local $ mkdir volttron-copier $ cd volttron-copier $ python3 -m venv env-copier $ source env-copier/bin/activate

(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $

Install copier

(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip install copier
Collecting copier
  Downloading copier-7.0.1-py3-none-any.whl (34 kB)
Collecting dunamai>=1.7.0
  Downloading dunamai-1.15.0-py3-none-any.whl (23 kB)
Collecting jinja2>=3.1.1
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
...

check copier version

(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip list | grep copier
copier                 7.0.1

Follow VOLTTRON/copier-poetry-volttron-agent README.md to generate a new agent-template

(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ copier "https://github.com/VOLTTRON/copier-poetry-volttron-agent.git" ./EmptyFolder
🎤 Your project name
   volttron_copier_agent
🎤 Your project description

🎤 Your full name
   Kefei
🎤 Your email
   kefei.mo@pnnl.gov
🎤 Your username (e.g. on GitHub)
   kefeimo
🎤 Your repository provider
   github.com
🎤 Your repository namespace
   [[ author_username ]]kefeimo
🎤 Your repository name
   [[ project_name|lower|replace('_', '-')|replace(' ', '-') ]]volttron_copier_a
gent
🎤 The name of the person/entity holding the copyright
   [[ author_fullname ]]Kefei Mo
🎤 The email of the person/entity holding the copyright
   [[ author_email ]]kefei.mo@pnnl.gov
🎤 The copyright date
   2023
🎤 Your project's license
   Apache License 2.0
🎤 Your Python package distribution name (for `pip install NAME`)
   [[ project_name|lower|replace('_', '-')|replace(' ', '-') ]]volttron-copier-a
gent
🎤 Your Python package import name (for `import NAME` in Python code)
   [[ project_name|lower|replace('-', '_')|replace(' ', '_') ]]volttron_copier_a
gent
🎤 Main class name for your agent
   [[ project_name|capitalize|replace('-', '_')|replace(' ', '_') ]]VolttronCopi
erAgent
🎤 Your CLI name if any (for use in the shell)
   [[ project_name|lower|replace('_', '-')|replace(' ', '-') ]]
🎤 Do you want to use pre-commit?
   No

Copying from template version 0.3.0b0.post15.dev0+3b3315f
    create  .
    create  config
    create  scripts
    create  scripts/post_generation.py
    create  .gitlab-ci.yml
    create  .github
    create  .github/workflows
    create  .github/workflows/code_analysis.yml
    create  .github/workflows/publish_to_pypi.yml
    create  .github/workflows/create_release.yml
    create  .github/workflows/ci.yml
    create  .github/ISSUE_TEMPLATE
    create  .github/ISSUE_TEMPLATE/feature_request.md
    create  .github/ISSUE_TEMPLATE/bug_report.md
    create  [[_copier_conf.answers_file]]
    create  CHANGELOG.md
    create  src
    create  src/[[ python_package_import_name ]]
    create  src/[[ python_package_import_name ]]/__init__.py
    create  src/[[ python_package_import_name ]]/agent.py
    create  mypy.ini
    create  CONTRIBUTING.md
    create  tests
    create  tests/test_cli.py
    create  tests/conftest.py
    create  docs
    create  docs/source
    create  docs/source/conf.py
    create  docs/source/index.rst
    create  docs/Makefile
    create  LICENSE
    create  .gitignore
    create  README.md
    create  pyproject.toml
    create  .pre-commit-config.yaml

 > Running task 1 of 1: python scripts/post_generation.py
Project successfully generated!

Observed (unexpected) output and the desired (expected) output

generated codebase scoffolding

Within the `src/[[ python_package_import_name ]]/agent.py

Produced python script with syntax error

(Optional) Full generated file as a tar/zip

EmptyFolder.zip

kefeimo commented 1 year ago

Ubuntu 20.04 Python 3.8.10

Prompts look fine, just doesn't use any of the input

copier "https://github.com/VOLTTRON/copier-poetry-volttron-agent.git" ./EmptyFolder

@acedrew Thanks for providing the info. I might have reproduced the issue you mentioned but still would like to clarify the following:

How did you get the error? Was it by following OLTTRON/copier-poetry-volttron-agent README.md
- If not, can you describe the steps to get the error? (Please see bug-reporting template in a similar scenario
Can you describe how did you install "copier" and confirm what's the installed version? (e.g., using pip list | grep copier or copier -v)
Can you point out the observed (unexpected) issue and desired (expected) result.
- I can see you attached a screenshot of "agent.py". Is this file generated from volttron-copier? Can you mark the part that is wrong?

Thank you.

@craig8 I suspect the volttron-copier workflow is broken for copier==7. Can you confirm what is the workable copier version for volttron-copier template at your side? Originally I have tested with copier==5.1.0 (I describe the reproducing process at https://github.com/VOLTTRON/copier-poetry-volttron-agent/issues/63#issuecomment-1416863895)

acedrew commented 1 year ago

Yes, I followed either the readme in this repo, or the readme in the Copier repo for steps that were missing, including using pipx for installation. It's 7.0.1 It's literally exactly the same as your example. None of the Class or module names were interpolated.

kefeimo commented 1 year ago

kefei@ubuntu-20-ws ~/project-local/volttron-copier $ source env-copier/bin/activate
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip list | grep copier
copier                 7.0.1  
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip install copier==5.1.0
Collecting copier==5.1.0
  Downloading copier-5.1.0-py3-none-any.whl (22 kB)
Requirement already satisfied: plumbum<2.0.0,>=1.6.9 in ./env-copier/lib/python3.8/site-packages (from copier==5.1.0) (1.8.1)
Collecting regex<2021.0.0,>=2020.6.8
  Downloading regex-2020.11.13-cp38-cp38-manylinux2014_x86_64.whl (738 kB)
   ...>copier==5.1.0) (2.1.2)
Installing collected packages: regex, pyyaml, pathspec, pyparsing, packaging, jinja2, copier
  Attempting uninstall: pyyaml
    Found existing installation: PyYAML 6.0
    Uninstalling PyYAML-6.0:
      Successfully uninstalled PyYAML-6.0
  Attempting uninstall: pathspec
    Found existing installation: pathspec 0.11.0
    Uninstalling pathspec-0.11.0:
      Successfully uninstalled pathspec-0.11.0
  Attempting uninstall: packaging
    Found existing installation: packaging 23.0
    Uninstalling packaging-23.0:
      Successfully uninstalled packaging-23.0
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 3.1.2
    Uninstalling Jinja2-3.1.2:
      Successfully uninstalled Jinja2-3.1.2
  Attempting uninstall: copier
    Found existing installation: copier 7.0.1
    Uninstalling copier-7.0.1:
      Successfully uninstalled copier-7.0.1
Successfully installed copier-5.1.0 jinja2-2.11.3 packaging-20.9 pathspec-0.8.1 pyparsing-3.0.9 pyyaml-5.4.1 regex-2020.11.13
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip list | grep MarkupSafe
MarkupSafe             2.1.2     
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip list | grep copier
copier                 5.1.0     
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip install MarkupSafe<2.1.0
bash: 2.1.0: No such file or directory
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ pip install MarkupSafe==2.0.1
Collecting MarkupSafe==2.0.1
  Downloading MarkupSafe-2.0.1-cp38-cp38-manylinux2010_x86_64.whl (30 kB)
Installing collected packages: MarkupSafe
  Attempting uninstall: MarkupSafe
    Found existing installation: MarkupSafe 2.1.2
    Uninstalling MarkupSafe-2.1.2:
      Successfully uninstalled MarkupSafe-2.1.2
Successfully installed MarkupSafe-2.0.1
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ copier "https://github.com/VOLTTRON/copier-poetry-volttron-agent.git" ./EmptyFolder2

Your project name
project_name? Format: str
🎤 [None]: volttron_copier_agent

Your project description
project_description? Format: str
🎤 [None]: 

Your full name
author_fullname? Format: str
🎤 [None]: kefei

Your email
author_email? Format: str
🎤 [None]: kefei.mo@pnnl.gov

Your username (e.g. on GitHub)
author_username? Format: str
🎤 [None]: kefeimo

Your repository provider
repository_provider? Format: str
🎤
(1) github.com
(2) gitlab.com
Choice [1]: 1

Your repository namespace
repository_namespace? Format: str
🎤 [kefeimo]: 

Your repository name
repository_name? Format: str
🎤 [volttron-copier-agent]: 

The name of the person/entity holding the copyright
copyright_holder? Format: str
🎤 [kefei]: 

The email of the person/entity holding the copyright
copyright_holder_email? Format: str
🎤 [kefei.mo@pnnl.gov]: 

The copyright date
copyright_date? Format: str
🎤 [2022]: 2023

Your project's license
copyright_license? Format: str
🎤
(1) Academic Free License v3.0
(2) Apache License 2.0
(3) MIT License
(4) Mozilla Public License 2.0
(5) Microsoft Public License
(6) The Unlicense
Choice [2]: 

Your Python package distribution name (for `pip install NAME`)
python_package_distribution_name? Format: str
🎤 [volttron-copier-agent]: 

Your Python package import name (for `import NAME` in Python code)
python_package_import_name? Format: str
🎤 [volttron_copier_agent]: 

Main class name for your agent
python_agent_class_name? Format: str
🎤 [Volttron_copier_agent]: 

Your CLI name if any (for use in the shell)
python_package_command_line_name? Format: str
🎤 [volttron-copier-agent]: 

Do you want to use pre-commit?
use_precommit? Format: bool
🎤? [y/N] y

    create  config
    create  .gitlab-ci.yml
    create  .copier-answers.yml
    create  CHANGELOG.md
    create  mypy.ini
    create  CONTRIBUTING.md
    create  LICENSE
    create  .gitignore
    create  README.md
    create  pyproject.toml
    create  .pre-commit-config.yaml
    create  scripts/
    create  scripts/post_generation.py
    create  .github/
    create  .github/workflows/
    create  .github/workflows/code_analysis.yml
    create  .github/workflows/publish_to_pypi.yml
    create  .github/workflows/create_release.yml
    create  .github/workflows/ci.yml
    create  .github/ISSUE_TEMPLATE/
    create  .github/ISSUE_TEMPLATE/feature_request.md
    create  .github/ISSUE_TEMPLATE/bug_report.md
    create  src/
    create  src/volttron_copier_agent/
    create  src/volttron_copier_agent/__init__.py
    create  src/volttron_copier_agent/agent.py
    create  tests/
    create  tests/test_cli.py
    create  tests/conftest.py
    create  docs/
    create  docs/Makefile
    create  docs/source/
    create  docs/source/conf.py
    create  docs/source/index.rst

 > Running task 1 of 1: python scripts/post_generation.py
Project successfully generated!

(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ tree 
EmptyFolder/  EmptyFolder2/ env-copier/   
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ tree EmptyFolder2
EmptyFolder2
├── CHANGELOG.md
├── config
├── CONTRIBUTING.md
├── docs
│   ├── Makefile
│   └── source
│       ├── conf.py
│       └── index.rst
├── LICENSE
├── mypy.ini
├── pyproject.toml
├── README.md
├── scripts
├── src
│   └── volttron_copier_agent
│       ├── agent.py
│       └── __init__.py
└── tests
    ├── conftest.py
    └── test_cli.py

6 directories, 14 files
(env-copier) kefei@ubuntu-20-ws ~/project-local/volttron-copier $ cat EmptyFolder2/src/volttron_copier_agent/agent.py 
"""
Agent documentation goes here.
For a quick tutorial on Agent Development, see https://volttron.readthedocs.io/en/develop/developing-volttron/developing-agents/agent-development.html#agent-development
"""

...

def volttron_copier_agent(config_path, **kwargs):
    """
    Parses the Agent configuration and returns an instance of
    the agent created using that configuration.

    :param config_path: Path to a configuration file.
    :type config_path: str
    :returns: Volttron_copier_agent
    :rtype: Volttron_copier_agent
    """
    try:
        config = utils.load_config(config_path)
    except Exception:
        config = {}

    if not config:
        _log.info("Using Agent defaults for starting configuration.")

    setting1 = int(config.get('setting1', 1))
    setting2 = config.get('setting2', "some/random/topic")

    return Volttron_copier_agent(setting1, setting2, **kwargs)

...

kefeimo commented 1 year ago

@acedrew, I tested with copier==5.1.0 (with MarkupSafe==2.0.1 as a patch) and it worked. I recommend using pip to install copier rather than pipx. The relative commands are

source env-copier/bin/activate
pip install copier==5.1.0
pip install MarkupSafe==2.0.1
copier "https://github.com/VOLTTRON/copier-poetry-volttron-agent.git" ./EmptyFolder2

Also use which copier to make sure the installation is within the virtualenv, not the OS one (or the one installed by pipx.)

Please see https://github.com/VOLTTRON/copier-poetry-volttron-agent/issues/63#issuecomment-1419969779 for full steps

Let me know if you run into other issues.

craig8 commented 1 year ago

@acedrew would you be willing to be a tester on this pr: hot-fix: updated README to handle broken workflow in copier==7, advo… #64? if no I will do it later today/thursday.

acedrew commented 1 year ago

I would argue that since it's advised by the copier community to use pipx, we should just provide the hotfix instructions for pipx pipx install copier<6.0.0 pipx runpip install markup==2.0.1

Which worked for me

kefeimo commented 1 year ago

I would argue that since it's advised by the copier community to use pipx, we should just provide the hotfix instructions for pipx pipx install copier<6.0.0 pipx runpip install markup==2.0.1

Which worked for me

Thanks for the feedback @acedrew, the reason I adopted pip + virtual environment workflow is

pip is more widely used than pipx
virtual environment is one of the best practices Plus, we we look at the pipx documentation about its installation at Linux, it still relies on pip. In other words, we can compare the 2+1 copier installation workflow workflow 1: pipx workflow at OS level
1. pip install pipx
2. pipx install copier<6.0.0 workflow 1b: pipx workflow at virtualenv level
3. create a virtual env called env
4. (within env) pip install pipx
5. pipx install copier<6.0.0 workflow 2: pip workflow
6. create a virtual env called env
7. (within env) pip install copier==5.0.1

I prefer workflow 2, since

it follows python best practices to use virtual environment (vs. workflow 1),
and it doesn't require extra dependency of pipx (vs. workflow 1b)

Hope that makes sense.

acedrew commented 1 year ago

The point of pipx is to have a way of running non-project specific command line tools in a virtual environment, isolated from other command line tools that may have dependency conflicts. When you run pipx install copier, you're instantiating a dedicated virtual environment for copier that guarantees that the later markupsafe==2.x.x doesn't interfere with any other dependencies. So your point about virtual environments being best practice is moot.

On Fri, Feb 10, 2023, 11:00 AM kefeimo @.***> wrote:

I would argue that since it's advised by the copier community to use pipx, we should just provide the hotfix instructions for pipx pipx install copier<6.0.0 pipx runpip install markup==2.0.1

Which worked for me

Thanks for the feedback @acedrew https://github.com/acedrew, the reason I adopted pip + virtual environment workflow is

pip is more widely used than pipx

virtual environment is one of the best practices Plus, we we look at the pipx documentation about its installation https://pypa.github.io/pipx/installation/ at Linux, it still relies on pip. In other words, we can compare the 2+1 copier installation workflow workflow 1: pipx workflow at OS level

pip install pipx

pipx install copier<6.0.0 workflow 1b: pipx workflow at virtualenv level

create a virtual env called env

(within env) pip install pipx

pipx install copier<6.0.0 workflow 2: pip workflow

create a virtual env called env

(within env) pip install copier==5.0.1

I prefer workflow 2, since

it follows python best practices to use virtual environment (vs. workflow 1),

and it doesn't require extra dependency of pipx (vs. workflow 1b)

Hope that makes sense.

— Reply to this email directly, view it on GitHub https://github.com/VOLTTRON/copier-poetry-volttron-agent/issues/63#issuecomment-1426016231, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHOUB2PKKTLLU6Y2KTF2F3WWZQ2JANCNFSM6AAAAAAUQ3FXW4 . You are receiving this because you were mentioned.Message ID: @.***>

kefeimo commented 1 year ago

So your point about virtual environments being best practice is moot.

I don't think "your point about virtual environments being best practice is moot" should be used as an argument here, because pipx uses virtual enviornment underneath the hood, quote: "Virtual Environments will be installed to $PIPX_HOME/venvs. ..." from Docs - pipx. But if you think we should use pipx since the copier docs suggest so I can include that as well.

@craig8 , I cannot use copier with pipx install at the OS level, I encountered the following error. (I already created alias for python="python3") I would not feel comfortable to only include the instruction using pipx since I cannot reproduce a working example at my end. I appreicate it if I can get some guidance. Thank you.

@shwethanidd, fell free to make comments since you advocate pipx as well.

acedrew commented 1 year ago

@kefeimo maybe you misunderstood my point? "using virtual environments is best practices" is a moot point because both approaches use virtual environment. Try `alias python python3'

kefeimo commented 1 year ago

@kefeimo maybe you misunderstood my point? "using virtual environments is best practices" is a moot point because both approaches use virtual environment. Try `alias python python3'

@acedrew how do you install pipx to start with? Was it using pip as pipx suggested for Linux or some other method? If using pip, which pip is it? os level or isolated environment? The point is if it is at os level then it breaks the isolation principle, if is the latter, you are already using isolation tools then why add another layer of abstraction? ‘alias Python python3’ is not a bash syntax for Ubuntu 20. What shell are you using?

craig8 commented 1 year ago

@kefeimo based upon the documentation I have used

python3 -m pip install --user pipx
python3 -m pipx ensurepath

The first is for pipx and the second makes sure that ~/.local/bin is on the path.

craig8 commented 1 year ago

So a couple of things in this list of stuff.

Pipx should be at a user level not a per environment level, because its an application wrapper for utilities/applications. It is for those things that cross environmental boundaries in the management.
Copier should not need to be tied to a single agent virtual environment because it shouldn't ever be deployed by that same virtual environment.
Poetry should be installed at a user level, unless it is needed as a library that should be distributed with it.

From what I have read we need to figure out how we are going to update the README for copier such that we have copier available on the PATH and copier is > 6. I just did a test and did the pipx install copier and by default it does use 7.0.1.

craig8 commented 1 year ago

I think the readme should just include the two lines above for installing pipx and then installing copier from there. The pyenv stuff mentioned in the linked copier page is nice but I think the shortest path is best here. BTW I got the pipx install from https://pypa.github.io/pipx/

kefeimo commented 1 year ago

@craig8 python3 -m pip install --user pipx python3 -m pipx ensurepath still not working for me. I just spin-up a brand-new Ubuntu 22.04. Still the same complaint about python not found.

@acedrew, @shwethanidd Another question: if the user require different version of copier, say copier==7.0.1 and copier==5.1.0, how to achieve that. (For use case like 1. copier==5.1.0 for volttron-copier, copier==7.0.1 to other template? Sure we can always update the volttron-copier template to use the most updated copier version, but for hot-fix scenario, how does it work?)

kefeimo commented 1 year ago

So a couple of things in this list of stuff.

Pipx should be at a user level not a per environment level, because its an application wrapper for utilities/applications. It is for those things that cross environmental boundaries in the management.

Copier should not need to be tied to a single agent virtual environment because it shouldn't ever be deployed by that same virtual environment.

Poetry should be installed at a user level, unless it is needed as a library that should be distributed with it.

From what I have read we need to figure out how we are going to update the README for copier such that we have copier available on the PATH and copier is > 6. I just did a test and did the pipx install copier and by default it does use 7.0.1.

Is poetry needed? I searched for the whole codebase and it is not referenced.
I am not surprised "by default it does use 7.0.1." since it uses pip underneath the hood and install the most stable-released version.
"we need to figure out how we are going to update the README for copier such that we have copier available on the PATH and copier is > 6."
- I think @craig8 meant to say copier is < 6
- I was able to "have copier available on the PATH " but got the "python not found" complain. Would like to see a from-scratch recipe (from installing pipx and copier) to generate an agent-scaffolding. (still I think using venv is more standardized, and the path is available once activated/sourced.)

kefeimo commented 1 year ago

Another question: if the user require different version of copier, say copier==7.0.1 and copier==5.1.0, how to achieve that. Answered by Allow multiple versions of Python package binaries

acedrew commented 1 year ago

Copier is not related to the virtualenv for the project, and is not concerned with the Python version of the template it's populating, so it doesn't need to be running in the same python environment as the project, in fact it's undesirable.

On Fri, Feb 10, 2023, 5:16 PM kefeimo @.***> wrote:

Another question: if the user require different version of copier, say copier==7.0.1 and copier==5.1.0, how to achieve that. Answered by Allow multiple versions of Python package binaries https://github.com/pypa/pipx/issues/241#top

— Reply to this email directly, view it on GitHub https://github.com/VOLTTRON/copier-poetry-volttron-agent/issues/63#issuecomment-1426410754, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHOUB335IXYNUEQ325JJJ3WW245VANCNFSM6AAAAAAUQ3FXW4 . You are receiving this because you were mentioned.Message ID: @.***>

craig8 commented 1 year ago

Poetry eventually will be required for Voltron core.

kefeimo commented 1 year ago

Copier is not related to the virtualenv for the project, and is not concerned with the Python version of the template it's populating, so it doesn't need to be running in the same python environment as the project, in fact it's undesirable.

@acedrew I don't think copier is related to the virtualenv for the project either. My apology if I ever indicated that. I thought the reason that I brought up virtual environment and isolation is pipx can be less familiar to users and I had problem installing copier using the pipx workflow. But if users have strong preference using pipx then I guess I made wrong assumption.

kefeimo commented 1 year ago

@craig8, what would be the disadvantages to have poetry as a virtualenv level dependency (either virtualenv alone, or virtualenv + user level)?

needs extra steps to use poetry as CLI tool (i.e., to activate the venv?) then "alias poetry="/bin/poetry"
I believe virtualenv resolve to the first, so the pyproject.toml should still behave as expected.

eclipse-volttron / copier-poetry-volttron-agent