ydf0509 / funboost

pip install funboost,python全功能分布式函数调度框架,funboost的功能是全面性重量级,用户能想得到的功能99%全都有;funboost的使用方式是轻量级,只有@boost一行代码需要写。支持python所有类型的并发模式和一切知名消息队列中间件,支持如 celery dramatiq等框架整体作为funboost中间件,python函数加速器,框架包罗万象,用户能想到的控制功能全都有。一统编程思维,兼容50% python业务场景,适用范围广。只需要一行代码即可分布式执行python一切函数,99%用过funboost的pythoner 感受是 简易 方便 强劲 强大,相见恨晚 。
Apache License 2.0
682 stars 135 forks source link

qps设置了5但是好像并不精准 #86

Closed User-Clb closed 1 year ago

User-Clb commented 1 year ago

import time from funboost import boost, BrokerEnum

@boost('test_queue', broker_kind=BrokerEnum.REDIS, qps=5, is_using_distributed_frequency_control=True) def ff(x, y): import os time.sleep(2) print(os.getpid(), x, y)

if name == 'main': ff.clear() for i in range(1000): ff.push(i, y=i * 2) ff.multi_process_start(2)

log: 2023-09-29 21:35:02 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 100 200 2023-09-29 21:35:02 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 0 0 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 1 2 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 101 202 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 2 4 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 3 6 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 4 8 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 102 204 2023-09-29 21:35:03 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 5 10 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 103 206 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 6 12 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 7 14 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 104 208 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 8 16 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 9 18 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 10 20 2023-09-29 21:35:04 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 105 210 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 11 22 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 106 212 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 12 24 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 13 26 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 107 214 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 14 28 2023-09-29 21:35:05 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 15 30 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 108 216 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 16 32 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 17 34 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 109 218 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 18 36 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 19 38 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 110 220 2023-09-29 21:35:06 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 20 40 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 21 42 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 111 222 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 22 44 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 23 46 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 24 48 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 3008 112 224 2023-09-29 21:35:07 "c:\Users\CC\Desktop\Test_1\10.py:8" -ff-[print]- 12688 25 50

User-Clb commented 1 year ago

@ydf0509 是我哪里使用错误了吗?

ydf0509 commented 1 year ago

有点笨,你只发布10个怎么观察,你push1000个任务观察最好。

你设置qps就不需要设置并发数量啊,自动调整并发数量的不需要手动指定并发数量。

ydf0509 commented 1 year ago

你设置qps,根本就不用自己根据函数耗时调线程数量啊,多此一举。

User-Clb commented 1 year ago

@ydf0509 重新调整了代码,qps设置为5,这个qps控频好像不是很准确 屏幕截图 2023-09-29 213654

User-Clb commented 1 year ago

@ydf0509 作者你好,请问这个框架有详细一点的API文档吗?

ydf0509 commented 1 year ago

你要看github readme啊,不是说了readthedocs文档吗,你连readme都没看嘛。

ydf0509 commented 1 year ago

我建议你设置20到100qps,funboost这已经比celery的qps控频精确很多了。

ydf0509 commented 1 year ago

文档托管在readthedocs了。

User-Clb commented 1 year ago

听取了你的建议设置20到100qps范围,尝试设置qps为30后确实效果好很多

ydf0509 commented 1 year ago

qps太小时候,就别开多进城了,这样qps就准了,qps才5开多进城浪费内存

User-Clb commented 1 year ago

是的,体验到funboost是一个强大的框架了,他会自治

User-Clb commented 1 year ago

是的,体验到funboost是一个强大的框架了,他会自治

User-Clb commented 1 year ago

@ydf0509 https://github.com/ydf0509/funboost/issues/87 这个问题依然困扰着我,push1000条消息,被消费的却总是低于push量,控制台也没有消费失败

ydf0509 commented 1 year ago

你自己调用了clear了,清除消息了,

User-Clb commented 1 year ago

@ydf0509 代码先执行的clear,后执行文件2的push,并不是push过程中或push后再执行的clear