xorbitsai / xoscar

Python actor framework for heterogeneous computing.
https://xoscar.dev
Apache License 2.0
89 stars 21 forks source link

Support fury as an optional serializer #78

Open codingl2k1 opened 1 year ago

codingl2k1 commented 1 year ago

What do these changes do?

Related issue number

import xorbits
import time
from xorbits.remote import spawn
from xoscar.serialization.core import FurySerializer
from dataclasses import dataclass
from typing import List

@dataclass
class Foo:
    f1: int
    f2: int
    f3: float
    f4: int
    f5: int
    f6: List[int]

FurySerializer.register(Foo)

import logging

logging.basicConfig(level=logging.DEBUG)

if __name__ == "__main__":
    xorbits.init(config={"third_party_modules":["t1"]})

    def f1(x):
        return x

    foos = [Foo(f1=5000, f2=2 ** 32, f3=1 / 3, f4=100, f5=200, f6=[11, 1234]) for i in range(500)]

    start = time.time()
    results = []
    for _ in range(100):
        r1 = spawn(f1, foos)
        results.append(r1)

    xorbits.run(results)

    print(f"cost: {time.time() - start}")

This test code may be crash when fury deserializing, but if it does not crash, the performance is better than Mars default serializers.

Use fury(if not crash): 9.764425039291382 Mars default: 13.590802669525146

Fixes #xxxx

Check code requirements

codecov[bot] commented 1 year ago

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (2a041ae) 89.12% compared to head (d43d8f4) 88.98%.

Files Patch % Lines
python/xoscar/serialization/pyfury.py 83.33% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #78 +/- ## ========================================== - Coverage 89.12% 88.98% -0.15% ========================================== Files 48 48 Lines 4010 4012 +2 Branches 764 764 ========================================== - Hits 3574 3570 -4 - Misses 351 356 +5 - Partials 85 86 +1 ``` | [Flag](https://app.codecov.io/gh/xorbitsai/xoscar/pull/78/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xorbitsai) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/xorbitsai/xoscar/pull/78/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xorbitsai) | `88.80% <83.33%> (-0.15%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xorbitsai#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

chaokunyang commented 10 months ago

Is there any things I can help for this PR?

codingl2k1 commented 10 months ago

Is there any things I can help for this PR?

fury Python is hard to use, could you help fix the crash issue?