search

Tencent / Firestorm

Firestorm is a Remote Shuffle Service, and provides the capability for Apache Spark and Apache Hadoop MapReduce applications to store shuffle data on remote servers
Other
252 stars 72 forks source link

[Bugfix] Fix MR don't have remote storage information when we use dynamic conf and MEMORY_LOCALE_HDFS storageType #195

Closed jerqi closed 2 years ago

jerqi commented 2 years ago

What changes were proposed in this pull request?

We should aquire the storageType from extraConf.

Why are the changes needed?

If we don't have this patch, MR don't work when we use dynamic conf and MEMORY_LOCALE_HDFS storageType.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual test

jerqi commented 2 years ago

Later we will add remote storage information check in StorageManager