While this will definitely be tricky to pull off, there's every reason to do this
CloudFormation has been stated by AWS support to be the inferior way to do IaC, and that terraform should be used instead
Terraform is more scalable and understood by a wider audience
Terraform has more features to allow some of the more complex aspects of deployment to be automated
Lambda functions can be automated in zipping and uploading code changes
Some functions in the infrastructure have no way to be defined in IaC from what I could find (e.g. certificate manager, some attributes in EC2, sharing values across stacks, etc)
Terraform has the import operation which means this transition could potentially be seamless.
Terraform is understood by more people on the team.
Set as "really nice to have" but the effort involved means that it isn't a priority at the moment. It may become an issue later if we wish to scale or automate further.
While this will definitely be tricky to pull off, there's every reason to do this