ray-project / rayfed

A multiple parties joint, distributed execution engine based on Ray, to help build your own federated learning frameworks in minutes.
https://rayfed.readthedocs.io
Apache License 2.0
91 stars 20 forks source link

not init kv until `fed.init` called #133

Closed NKcqx closed 1 year ago

NKcqx commented 1 year ago

Fed will try to init the internal_kv when importing the "compatible_utils" module. This will cause problem in client-server mode, because the import action comes first before the call ofray.init, and at that time, the flag ray._private.client_mode_hook.is_client_mode_enabled that internal_kv relies on is not "True" as expected.

This PR fixes it by only initializing internal_kv when explicitly calling _init_internal_kv and only init once.

fengsp commented 1 year ago

We run the following code in fed.config module:

        compatible_utils._init_internal_kv()
        compatible_utils.kv.initialize()

Is this intentional?

NKcqx commented 1 year ago

We run the following code in fed.config module:

        compatible_utils._init_internal_kv()
        compatible_utils.kv.initialize()

Is this intentional?

Yes, fed.config's "getter", i.e. get_cluster_config and get_job_config, need to access the internal_kv. It's indeed a little weird putting a dynamic action in a static-like config file, should consider refactoring these kinds of codes in another PR.

fengsp commented 1 year ago

initialize

If initialize called multiple times is ok, LGTM