Deferred values are not resolved before eval_generate

Describe the Bug

Type & providers that implement eval_generate to create child resource may be called before the parent deferred resource is resolved. This was first reported in https://github.com/puppetlabs/puppetlabs-concat/pull/789 where the concat parent resource generates concat_fragments.

Commit a76f49853996fe79766c279e4174f819e9839467 made it possible to lazily resolve deferred resources. The behavior was guarded by the preprocess_deferred=true setting. In puppet8, the default value was changed to false, so deferred resources are no longer preprocessed.

The intention of the change was to allow dependencies of a deferred resource to be managed prior to the deferred resource. For example, from @bastelfreak in https://puppetcommunity.slack.com/archives/CFD8Z9A4T/p1700590993590929

gitlab_runner module uses a deferred function to register itself to the gitlab server. the gitlab runner has a local firewall for outgoing traffic. so we have to apply puppet code to modify the firewall before the function is executed

However, as a result of this change, type and provider lifecycle methods (such as validation, autorequires, prefetch, generate and eval_generate) may be called with deferred resources still in the catalog, since those calls occur after the catalog is created, but before the deferred resource is evaluated.

Expected Behavior

I would expect the eval_generate method to be called after deferred values in the resource have been resolved, so that DeferredValue objects don't leak into custom types/providers.

Steps to Reproduce

# rpm -q puppet-agent
puppet-agent-8.3.1.86.g719cf6147-1.el7.x86_64
# /opt/puppetlabs/puppet/bin/puppet module install puppetlabs-concat
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppetlabs-concat (v9.0.0)
  └── puppetlabs-stdlib (v9.4.1)
# cat <<END > concat.pp
concat { '/tmp/deferred_file': }
concat::fragment { '1':
  target  => '/tmp/deferred_file',
  content => Deferred('md5', ['test']),
}
END
# puppet apply concat.pp
Warning: Deferred function md5 has no return_type, unable to guarantee value type during compilation.
   (file & line not available)
Notice: Compiled catalog for papal-mandrel.delivery.puppetlabs.net in environment production in 0.07 seconds
Notice: /Stage[main]/Main/Concat[/tmp/deferred_file]/File[/tmp/deferred_file]/ensure: defined content as '{sha256}276179e40d8de773d3819996cd861446eeeef83fb607df4e72a3768c3390c1d0'
Notice: Applied catalog in 0.01 seconds
# cat /tmp/deferred_file 
#<Puppet::Pops::Evaluator::DeferredValue:0x00007f8c92dba4b8>

Environment

This affects puppet 8 (by default) and puppet 7 (when running with preprocess_deferred=false).

Additional Context

https://github.com/puppetlabs/puppet/pull/8902 https://github.com/puppetlabs/puppet/pull/9152

puppetlabs / puppet