Update parallel_runner.py

[noklam]

Description

Development notes

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• [ ] Read the contributing guidelines
• [ ] Signed off each commit with a Developer Certificate of Origin (DCO)
• [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• [ ] Updated the documentation to reflect the code changes
• [ ] Added a description of this change in the RELEASE.md file
• [ ] Added tests to cover my changes
• [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

Find the seemingly obsoleted MemoryDataset used in ParallelRunner, remove and see CI agree.

[noklam]

Closed:

https://docs.python.org/3/library/multiprocessing.html#customized-managers

The complete story is that:

• ParallelRunnerManager inherit from SyncManager
• MemoryDataset is registered as a class method to ParallelRunnerManager
• The "shared memory datast" is really just MemoryDataset living in the ParallelRunnerManager
• Because it is lived in the SyncManager object, it can be shared across process.

In a way SharedMemoryDataset is a special wrapper dataset that coupled with the MemoryDataset.

[astrojuanlu]

Can the branch be deleted? (Didn't want to do it myself just in case)