preset-io / promptimize

Promptimize is a prompt engineering evaluation and testing toolkit.
Apache License 2.0
419 stars 32 forks source link

demjson issues on windows #8

Open hucker opened 1 year ago

hucker commented 1 year ago

Python Bytes Listener here:

I'm running this repo in Windows running promptimize with

promptimize --help

it generates the expected output, but when run promptimize run examples/ --output ./report.yaml

I get the following:

(venv)  chuck.b@MCX-C.B ~\source\repos\promptimize  venv 3.11.0  promptimize run examples/ --output ./report.yaml
💡 ¡promptimize! 💡
Traceback (most recent call last):                                                                            
  File "<frozen runpy>", line 198, in _run_module_as_main                                                     
  File "<frozen runpy>", line 88, in _run_code                                                                
  File "C:\Users\cbass\source\repos\promptimize\venv\Scripts\promptimize.exe\__main__.py", line 7, in <module>
  File "C:\Users\cbass\source\repos\promptimize\venv\Lib\site-packages\click\core.py", line 1130, in __call__ 
    return self.main(*args, **kwargs)                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                         
  File "C:\Users\cbass\source\repos\promptimize\venv\Lib\site-packages\click\core.py", line 1055, in main     
    rv = self.invoke(ctx)                                                                                     
         ^^^^^^^^^^^^^^^^                                                                                     
  File "C:\Users\cbass\source\repos\promptimize\venv\Lib\site-packages\click\core.py", line 1657, in invoke   
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                   
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                    
  File "C:\Users\cbass\source\repos\promptimize\venv\Lib\site-packages\click\core.py", line 1404, in invoke   
    return ctx.invoke(self.callback, **ctx.params)                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                            
  File "C:\Users\cbass\source\repos\promptimize\venv\Lib\site-packages\click\core.py", line 760, in invoke    
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cbass\source\repos\promptimize\promptimize\cli.py", line 94, in run
    uses_cases = discover_objects(path, BasePromptCase)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cbass\source\repos\promptimize\promptimize\crawler.py", line 45, in discover_objects
    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\cbass\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\cbass\source\repos\promptimize\examples\python_examples.py", line 68, in <module>
    lc_template = PromptTemplate(
                  ^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for PromptTemplate
__root__
  Invalid prompt schema; check for mismatched or missing input parameters. "Extra variables: {'format_instructions'}" (type=value_error)

I suspect that this is related to demjson.py not pip installing. The above code makes an effort to use demjson3.py (along with edits to reference demjson3.py), but it still fails.

paulz commented 1 year ago

Here is what helped me to workaround this issue on macOS:

https://github.com/dmeranda/demjson/issues/43#issuecomment-1681842851