[Task]: Improve python statecache

[AnandInguva]

What needs to happen?

Initially for python sdk, we will enable the statecache size from 0 MB to 100 MB. Then there are some improvements that could be made on the statecache. For example,

• auto size the cache based upon amount of memory dedicated to docker container
• add pipeline options to allow users to control % of memory for cache (with default of 20%) instead of relying on an experiment
• cache beyond the first page
• have a write through cache for user state

The Java implementation for the cache is in: https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/Caches.java And most of the caching complexity is within: https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java With the views over these caches doing specific view level operations (e.g. merging old view of data with in-memory updates). Generally understanding the code in https://github.com/apache/beam/tree/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state should provide most answers.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• [ ] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner

[AnandInguva]

cc: @tvalentyn

[AnandInguva]

AsIter view_fn

Iterable might look one element at a time and this could be more for the side input cache on the GCS bucket?

• https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_02_55-13940213406894669659
• Pipeline scaled up to ~ 47 workers for a simple job.

AsList view fn List materializes so we wouldn’t need too many reads from the side input cache at GCS bucket?

• https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_03_00-15374614790418733789
• Pipeline completed as expected

For AsIter with state_cache_size=100 mb,

• https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_23_37-9771630946918171566
• With state cache enabled, this completed as expected since side input gets cached.

[tvalentyn]

State cache was enabled in https://github.com/apache/beam/issues/28770 .