special character issues when dealing with openzepplin

OliverLaiGHong commented 4 months ago

Environment information

OS: linux (Ubuntu 20.04.6 LTS
Python Version: 3.11
ape and plugin versions:

$ ape --version
0.7.19

$ ape plugins list
Installed Plugins
  alchemy      0.7.2
  ens          0.7.1
  etherscan    0.7.2
  foundry      0.7.4
  hardhat      0.7.2
  infura       0.7.3
  polygon      0.7.2
  solidity     0.7.3
  template     0.7.0
  tokens       0.7.1
  vyper        0.7.1

Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

$ cat ape-config.yaml
name: play_ape
plugins:
  - name: polygon
  - name: ape-etherscan
  - name: foundry

default_ecosystem: polygon

polygon:
  default_network: mainnet
  mainnet:
    default_provider: geth

foundry:
  host: auto
  request_timeout: 20
  fork_request_timeout: 600
  fork:
    polygon:
      mainnet:
        upstream_provider: geth

geth:
  polygon:
    mainnet:
      uri: http://localhost:8545

dependencies:
  - name: OpenZeppelin
    github: OpenZeppelin/openzeppelin-contracts
    version: 5.0.1

solidity: 
  version: 0.8.23
  import_remapping:
    - "@openzeppelin=OpenZeppelin/5.0.1"

compile:
  include_dependencies: true

What went wrong?

I tried using solidity v0.8.23 and v0.8.25, and openzepplin v5.0.1 and v5.0.2, i keep getting the special character issues. when I removed everything the needs openzepplin, it got no problems.

Please include information like:

what command you ran $ ape compile

the code that caused the failure and the full output of the error received: after compiling all my own contracts and interfaces I obtained from dexes' websites, as soon as it starts working on openzepplin dependencies, this error occurs:

Traceback (most recent call last):                                                                                                                                                               
File "/root/py_envs/ape/bin/ape", line 8, in <module>                                                                                                                                          
sys.exit(cli())                                                                                                                                                                              
         ^^^^^                                                                                                                                                                               
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1157, in __call__                                                                                                    
return self.main(*args, **kwargs)                                                                                                                                                            
       ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                            
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1078, in main                                                                                                        
rv = self.invoke(ctx)                                                                                                                                                                        
     ^^^^^^^^^^^^^^^^                                                                                                                                                                        
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/_cli.py", line 87, in invoke                                                                                                          
return super().invoke(ctx)                                                                                                                                                                   
       ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                   
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1688, in invoke                                                                                                      
return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                      
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_compile/_cli.py", line 54, in cli
contract_types = cli_ctx.project_manager.load_contracts(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/manager.py", line 765, in load_contracts
manifest = self.local_project.create_manifest(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 193, in create_manifest
compiled_contract_types = self._compile(project_sources)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 270, in _compile
return _compile_sources(project_sources)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 259, in _compile_sources
return self.compiler_manager.compile(srcs_to_compile)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/compilers.py", line 172, in compile
compiled_contracts = compiler.compile(
                     ^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 518, in compile
input_jsons = self.get_standard_input_json(contract_filepaths, base_path=base_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 479, in get_standard_input_json
files_by_solc_version = self.get_version_map(contract_filepaths, base_path=base_path)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 761, in get_version_map
self.get_import_remapping(base_path=base_path)
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 303, in get_import_remapping
self._add_dependencies(manifest, sub_contracts_cache, builder)
File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 330, in _add_dependencies
cached_source.write_text(str(src.content))
File "/usr/lib/python3.11/pathlib.py", line 1079, in write_text
return f.write(data)
       ^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 1078: ordinal not in range(256)

How can it be fixed?

Sorry. I have no idea.

linear[bot] commented 4 months ago

APE-1744 special character issues when dealing with openzepplin

fubuloubu commented 4 months ago

I had noticed that interfaces/IERC4626.sol is using a utf-8 special character for quotes in a comment somewhere, not sure if this is the problem but that file has been like that for 2+ years so I wouldn't think so

OliverLaiGHong commented 4 months ago

@fubuloubu Actually, these utf-8 characters are seen across multiple files. i have tried to remove them but it was proven impractical. however, Ape was able to handle them about a month ago.

antazoey commented 3 months ago

sorry - accidentally closed

antazoey commented 3 months ago

I can reproduce this problem by doing:

path.write_text("“", encoding="latin-1")

so that is why specifying the encoding as utf8 everywhere fixes the problem. (see attached 2 issues)

also - this is no longer a problem in this specific use-case anyway because we are using shutil.copytree() now as of https://github.com/ApeWorX/ape/pull/2123

so this issue will be closed in every way once the 2 attached PRs merge.

ApeWorX / ape