If that ever becomes a problem, ripping it out is very easy at least in this case. You're missing out on the benefits based on an eventuality in the mean time.
This action only makes whatever cargo/rustc put on disk in ~/.cargo and ./target available on the next run, so unless it starts actually breaking those files, I'd consider any such issue a bug in cargo/rustc.
Reduces build time by 0.5-1.5 minutes depending on the job.