Clarinet integrate doesn't work with [[project.requirements]]

[mcintyre94]

Moving discussion from Discord: https://discord.com/channels/621759717756370964/839633619261456444/973941248475938816

For anyone not familiar, [[project.requirements]] is a field in Clarinet.toml that allows 'pulling in' a testnet or mainnet contract that can be used as a dependency in contracts. For example, if we have the mainnet SIP-010 contract in that field:

# Clarinet.toml
[[project.requirements]]
contract_id = "SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard"

Then our contracts can reference the trait, eg. (use-trait ft-trait 'SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard.sip-010-trait)

Most clarinet commands (eg. check, console) will work with contracts using this feature. But clarinet integrate doesn't work correctly - any contract using a requirement will fail to deploy. This is because in the local node there is no contract with eg. SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard. Currently the only way to work around this is to deploy the contract in the clarinet project, and modify the code manually to use the locally deployed one instead. This must of course be changed back to deploy, we can't run both clarinet integrate and clarinet contracts publish on the same code.

Ideally we want to run clarinet integrate on the code that will be deployed, so we want to be able to include eg. (use-trait ft-trait 'SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard.sip-010-trait) and have it work in clarinet integrate.

Two solutions were discussed:

• The stacks node in clarinet integrate could be modified to allow deploying a contract as any address. Then if we have SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard in requirements, then the local node would deploy that code as SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE, and our code using that would work correctly

• We could instead deploy the code from the requirements as the deployer. The contracts could then be transparently modified (as part of clarinet integrate and only in memory) to strip out the address and use the local deployment. Eg. we could replace (use-trait ft-trait 'SP3FBR2AGK5H9QBDH3EEN6DF8EK8JY7RX8QJ5SVTE.sip-010-trait-ft-standard.sip-010-trait) with (use-trait ft-trait .sip-010-trait-ft-standard.sip-010-trait) and then deploy.

My not super well informed view of these is:

• Option 1 sounds harder, and maybe harder to maintain as stacks changes over time?
• Option 1 is maybe scary, because it makes the stacks node less realistic
• Option 2 is also maybe scary because you're modifying the code that's being deployed. But this is way better than making the user do roughly the same thing manually each time they want to switch from integrate to deploy?
• Option 2 wouldn't be entirely transparent, the extra deploys + the contract modifications will all be visible in explorer
• Option 2 would fail if there was a contract with the same name in both requirements and local. Option 1 should be resilient to this. I think this is minor though.

[sabbyanandan]

@mcintyre94: Thank you for a detailed write-up! Much appreciated.

[lgalabru]

Thank you for opening this issue @mcintyre94, I like your sum-up on the different trade-offs. There could be a 3rd option: we could introduce a new kind of constructs in Clarinet, that would let us deal this specific scenario, but also others issues raised in the past.

We could tweak the parser or introduce a pre-processor in Clarinet, that would allow a construct like the following:

(contract-call? $MY_CONTRACT do-something) 

In your chain config / deployment files, you would specify that for a given environment, you want $MY_CONTRACT to be remapped with a contract-id of your choice, and you could have different values depending on the environment (Test/Devnet/Testnet/Mainnet) that you'd like to target.

Curious to collect some takes on this approach.

[mcintyre94]

Just to highlight a small change in behaviour, in Clarinet 0.3.0 contracts with these dependency issues don't get deployed at all in clarinet integrate. So you'll no longer see a deploy transaction fail, it'll just be missing.

[mcintyre94]

In case it helps anyone, I've written a Python script which will convert a Clarinet project ready to deploy w/ dev or mainnet dependencies and [[project.requirements]] into one that will work with clarinet integrate. Please make a git commit before running it so you can easily revert its changes if you try it out!

It will broadly do everything in option 2:

• Create each contract in [[project.requirements]] locally using the source code pulled from the dev/mainnet
• Update imports to use the local version
• Update the depends_on array for each updated contract
• Idempotent: you can run it multiple times, and it works even if you have the contract locally already

Notes:

• You need python3, with requests and toml libraries installed
• Changes to contracts are very precise, should just be one-line changes where you use the import
• Changes to Clarinet.toml are not precise, it'll move stuff around etc. Again, please make sure you can revert easily!

View Example

```shell $ clarinet check ✔ Syntax of 8 contract(s) successfully checked $ clarinet integrate (opens, then fails with error about deploy with requirements) $ python3 to-integrate.py Created file contracts/sip-010-trait-ft-standard.clar Created file tests/sip-010-trait-ft-standard_test.ts Updated Clarinet.toml with contract sip-010-trait-ft-standard Updated sip010-ft-trait.clar Updated magic-beans-token.clar Updated exchange-trait.clar Updated sip-010-trait-ft-standard.clar Updated cow-token.clar Updated liq-pool-trait.clar Updated other-exchange.clar Updated magic-stx-lp.clar Updated cow-stx-lp.clar Updated my-exchange.clar Added new dependencies to Clarinet.toml $ clarinet check ✔ Syntax of 9 contract(s) successfully checked $ clarinet integrate (successfully deploys all contracts) ```

Python script, place at the same level as Clarinet.toml

```python # to-integrate.py import os import requests import toml from dataclasses import dataclass import subprocess @dataclass class Requirement: address: str name: str @dataclass class NewDependency: contract_name: str requirement_name: str def make_requirement_locally(base_dir: str, requirement: Requirement): # fetch requirement source code if requirement.address.startswith('SP'): # mainnet url = f'https://stacks-node-api.mainnet.stacks.co/v2/contracts/source/{requirement.address}/{requirement.name}?proof=0' else: # testnet url = f'https://stacks-node-api.testnet.stacks.co/v2/contracts/source/{requirement.address}/{requirement.name}?proof=0' r = requests.get(url) contract_source: str = r.json()['source'] # Add the contract locally os.chdir(base_dir) os.environ["CLARINET_DISABLE_HINTS"] = "1" subprocess.call(["clarinet", "contract", "new", requirement.name]) # Set the local source to what we fetched with open(f'contracts/{requirement.name}.clar', 'w') as f: f.write(contract_source) def process_requirements(base_dir: str, requirements: list[Requirement]): contracts_dir = os.path.join(base_dir, 'contracts') contracts = [f for f in os.listdir(contracts_dir) if f.endswith('.clar')] new_dependencies: list[NewDependency] = [] for c in contracts: with open(os.path.join(contracts_dir, c), 'r+') as f: content = f.read() for r in requirements: to_replace = f"'{r.address}.{r.name}" replace_with = f'.{r.name}' if to_replace in content: new_dependencies.append(NewDependency(c.removesuffix('.clar'), r.name)) content = content.replace(to_replace, replace_with) # Overwrite the contract file with the replaced addresses f.seek(0) f.write(content) f.truncate() print(f'Updated {c}') # add new dependencies to Clarinet.toml with open(os.path.join(base_dir, 'Clarinet.toml'), 'r+') as f: content = f.read() settings = toml.loads(content) for d in new_dependencies: depends_on: list[str] = settings['contracts'][d.contract_name]['depends_on'] depends_on.append(d.requirement_name) depends_on = set(depends_on) settings['contracts'][d.contract_name]['depends_on'] = depends_on new_content = toml.dumps(settings) # write the new content f.seek(0) f.write(new_content) f.truncate() print('Added new dependencies to Clarinet.toml') def parse_requirement(requirement: dict[str, str]): address_name = requirement['contract_id'] address, name = address_name.split('.') return Requirement(address, name) def main(): # This is the directory the script is in, ie the path to defi-contracts/ base_dir = os.path.dirname(os.path.realpath(__file__)) with open(os.path.join(base_dir, 'Clarinet.toml'), 'r') as f: content = f.read() settings = toml.loads(content) requirements = settings['project']['requirements'] parsed_requirements = [parse_requirement(r) for r in requirements] # for each requirement, fetch its source + create a file for it + add it to Clarinet.toml for r in parsed_requirements: make_requirement_locally(base_dir, r) # for each contract, if it contains a requirement, replace it with local + add it to Clarinet.toml dependencies process_requirements(base_dir, parsed_requirements) if __name__ == '__main__': main() ```

[mcintyre94]

In your chain config / deployment files, you would specify that for a given environment, you want $MY_CONTRACT to be remapped with a contract-id of your choice, and you could have different values depending on the environment (Test/Devnet/Testnet/Mainnet) that you'd like to target.

I like this because it'd let you deploy to testnet and then mainnet without any code changes. Both solutions in the initial writeup would be limited to one or the other (in addition to local devnet), since you're hardcoding the address (which can only be on one or the other) into the contract.

[lgalabru]

Addressed in https://github.com/hirosystems/clarinet/pull/388