Ideas for scratchattach 2.0

TimMcCool commented 1 month ago

New / updated features in 2.0 (will be released in early October 2024)

These are only ideas for now. I don't know yet how many of these I'll actually implement. If you want to help by submitting a pull request, please choose the v2 branch as base.

New classes related to the Scratch Community:

[x] Session.connect_user_by_id
[x] Session.get_message_events -> MessageEvents class
[x] Session.create_project
[x] Session.create_remix
[x] scratchattach.get_comment, Session.connect_comment -> Comment class
_scratchattach.get_class, Session.connectclass -> Class class (for classrooms) cancled because there are no APIs for classrooms at all
[x] Update (or deprecate) existing forum classes and methods so they work without ScratchDB
[x] Session.connect_filterbot -> FilterBot class (a bot automatically deleting spam comments)
[x] ScratchActivity class for messages and other user activity items

New cloud classes:

[x] scratchattach.get_cloud, Session.connect_cloud -> Cloud class
[x] Cloud.connect_events, Cloud.get_events -> CloudEvents class
[x] Cloud.connect_recorder -> CloudRecorder class
[x] Cloud.connect_requests -> CloudRequests class (1)
[x] Cloud.connect_storage -> CloudStorage class (1)
[x] MultiCloudEvents, MultiCloudRequests, MultiCloudStorage classes (1)

New common methods:

[x] objectify (convert list of dicts to list of corresponding objects)
[x] api_request (performs a Scratch API request and automatically raises error based on API response)

Project JSON editing capabilities:

[x] ProjectJSON class with attributs like .user_agent, .block_count, .assets etc. (2)
[x] ProjectJSON.assets returns list of (new) Asset objects with attributes like .asset_id .type .download() etc. (2)
[x] list is also returned by Session.backpack() (2)
[x] maybe more features, like .add_var(), .convert_to_cloud_var() etc. (2)
[x] test backpack features

New features for cloud requests:

[x] packet loss prevention (issue #235 ) (1)
multithread cloud variable sets (not sure if possible) unnecessary because I found out how to set multiple vars simulatenously (1)
[x] better event loop: Event class of threading module ~~- add asyncio cloud requests~~ canceled
[x] debug CloudRequests, MultiEventHandler, CloudRequestsAsync and CloudStorage
[x] add .ping method to Cloud Requests (send data to Scratch without a priorly received event) (1)

Other stuff:

[x] TwCloudServer: a class for creating a websocket cloud server compatible with TurboWarp (3)
[x] ScratchAuth: a class with methods for authenticating a user's Scratch account in a app (3)
[x] Overhaul cloud variable methods for better error handling and prevent mass reconnects
[x] add method for parent_comment
[x] set multiple cloud vars simultaneously (maybe possible?)

TimMcCool commented 1 month ago

Generally, I'll try to increase consistencies in how scratchattach's classes work in v2.0.

I'll also make it so if you (for example) 1) run session.connect_user to get a User object and then 2) run user.followers() method to get the user's followers as list objects,

any methods requiring authentication will also work on the objects returned in step 2). Your session will be connected to all objects you get from an object where your session is connected, so you can flawlessly follow all users returned from user.followers, user.following etc. without having to run session.connect_user on them (the same will also apply to projects, studios etc.)

mas6y6 commented 1 month ago

Oh! I like this so this whats you been doing this entire time!

mas6y6 commented 1 month ago

You should add support for Multiple clients to run and if one goes down it can be redirected to an other client and you can also make it like a url that it requests things from one server or node like what I was planning to do with ScratchAttach Nexor

TheCommCraft commented 1 month ago

Are you using the profile comment api to get username by userid?

Am Mi., 25. Sept. 2024 um 19:13 Uhr schrieb mas6y6 @.***

:

You should add support for Multiple clients to run and if one goes down it can be redirected to an other client and you can also make it like a url that it requests things from one server or node like what I was planning to do with ScratchAttach Nexor

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2374680341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZUW54RVIY44WYXPPK3ZYLVMDAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUGY4DAMZUGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

TimMcCool commented 1 month ago

Are you using the profile comment api to get username by userid? Am Mi., 25. Sept. 2024 um 19:13 Uhr schrieb mas6y6 @. … : You should add support for Multiple clients to run and if one goes down it can be redirected to an other client and you can also make it like a url that it requests things from one server or node like what I was planning to do with ScratchAttach Nexor — Reply to this email directly, view it on GitHub <#242 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZUW54RVIY44WYXPPK3ZYLVMDAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUGY4DAMZUGE . You are receiving this because you are subscribed to this thread.Message ID: @.>

Yes. I'm planning to first check if the ID can be found in a DB run by PhilHub, and then use the profile comment API. (I'll put a warning saying that the API creates a message and shouldn't be used in large-scale)

TheCommCraft commented 1 month ago

You can just delete the comment to delete the message

Tim @.***> schrieb am Mi., 25. Sept. 2024, 21:02:

Are you using the profile comment api to get username by userid? Am Mi.,

Sept. 2024 um 19:13 Uhr schrieb mas6y6 @. … <#m596885099696569768> : You should add support for Multiple clients to run and if one goes down it can be redirected to an other client and you can also make it like a url that it requests things from one server or node like what I was planning to do with ScratchAttach Nexor — Reply to this email directly, view it on GitHub <#242 (comment) https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2374680341>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZUW54RVIY44WYXPPK3ZYLVMDAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUGY4DAMZUGE . You are receiving this because you are subscribed to this thread.Message ID: @.>

Yes. (I'll put a warning saying that the API creates a message and shouldn't be used in large-scale)

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2374928960, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZWEARA357VF54MU4OLZYMCCVAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUHEZDQOJWGA . You are receiving this because you commented.Message ID: @.***>

TimMcCool commented 1 month ago

The rate limit for posting comments isn't active when the new comment is deleted, this makes it possible to use the method for getting a user by id without having to implement a cooldown

TheCommCraft commented 1 month ago

So it's even better

Tim @.***> schrieb am Mi., 25. Sept. 2024, 23:17:

The rate limit for posting comments isn't active when the new comment is deleted, this makes it possible to use the method for getting a user by id without having to implement a cooldown

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2375278176, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZTYZ7UEC3VZFTYTS73ZYMSAHAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZVGI3TQMJXGY . You are receiving this because you commented.Message ID: @.***>

starlightsparks commented 1 month ago

I do beg your pardon; Does Session.create_project create a project? If so, is this not a bot-performed social action?

TheCommCraft commented 1 month ago

Maybe .push or .send would be better than .ping

TimMcCool commented 1 month ago

An option for using proxies will be part of v2.0

TheCommCraft commented 1 month ago

I saw a couple of bugs in the code, I'll make a fix pull request soon.

TimMcCool commented 1 month ago

~~the info in this post was wrong~~

TheCommCraft commented 1 month ago

Don't you mean parent_id?

Tim @.***> schrieb am Sa., 28. Sept. 2024, 15:08:

It appears that when replying to a project / studio comment, the commentee_id field has to be set or else the comment won't be loaded by the UI (it still can be accessed through the API tho)

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2380634634, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZSNSB3G2KJTTTEUBEDZY2S3LAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBQGYZTINRTGQ . You are receiving this because you commented.Message ID: @.***>

TimMcCool commented 1 month ago

Don't you mean parent_id? Tim @.> schrieb am Sa., 28. Sept. 2024, 15:08: … It appears that when replying to a project / studio comment, the commentee_id field has to be set or else the comment won't be loaded by the UI (it still can be accessed through the API tho) — Reply to this email directly, view it on GitHub <#242 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZSNSB3G2KJTTTEUBEDZY2S3LAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBQGYZTINRTGQ . You are receiving this because you commented.Message ID: @.>

I meant commentee_id. I thought the empty commentee_id was the reason my comments replying to a reply were not displayed, but the actual reason was that Scratch only loads comments replying to a top-level comment.

I just discovered that when commenting in the replies of a project / studio comment, Scratch will always use the comment_id of the top-level comment as parent_id - even when replying to a reply.

Scratch never uses the comment_id of reply as parent_id, and comments where the parent_id leads to a reply won't be loaded because Scratch only loads the first "layer" of replies.

Drumede commented 1 month ago

i genuinely cant wait for this version

TimMcCool commented 1 month ago

other.project_json_capabilities.ProjectBody (a class parsing and saving project json) is now fully working (I tested it with big projects like my among us remake), but there's still room for improvement (methods like sprite.add_var etc. could be added) and there may still be bugs. Feel free to contribute features / bug fixes

banddans commented 1 month ago

I know one thing to do is to connect multiple sockets, then you can bypass the cloud limit:

import scratchattach
import random
import time

class SubSession:
    def __init__(self):
        self.session = scratchattach.Session("nuhu", username="banddans")
        self.conn = self.session.connect_cloud("1234")

    def set_the_variable(self):
        self.conn.set_var("CLOUD", random.randint(0, 100))

sessions = []
for i in range(3):
    sessions.append(SubSession())

i = 0
while True:
    sessions[i%3].set_the_variable()
    time.sleep(1/30)
    i += 1

TimMcCool commented 1 month ago

I know one thing to do is to connect multiple sockets, then you can bypass the cloud limit:

I found out that you can set multiple cloud variables simultaneously (in one websocket message - cloud.set_vars(var_value_dict) will be added in v2.0), this probably makes it unnecessary to use multiple connections

TheCommCraft commented 1 month ago

Wasn't the limit 2 variables at a time?

Tim @.***> schrieb am Do., 3. Okt. 2024, 23:28:

I know one thing to do is to connect multiple sockets, then you can bypass the cloud limit:

import scratchattachimport randomimport time class SubSession: def init(self): self.session = scratchattach.Session("nuhu", username="banddans") self.conn = self.session.connect_cloud("1234")
def set_the_variable(self):
    self.conn.set_var("CLOUD", random.randint(0, 100))
sessions = []for i in range(3): sessions.append(SubSession()) i = 0while True: sessions[i%3].set_the_variable() time.sleep(1/30) i += 1

I found out that you can set multiple cloud variables simultaneously (in one websocket message - cloud.set_vars(var_value_dict) will be added in v2.0), this probably makes it unnecessary to use multiple connections

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2392370162, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZTIKXFGABGV65EQVZDZZWZIZAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJSGM3TAMJWGI . You are receiving this because you commented.Message ID: @.***>

banddans commented 1 month ago

I know one thing to do is to connect multiple sockets, then you can bypass the cloud limit:

I found out that you can set multiple cloud variables simultaneously (in one websocket message - cloud.set_vars(var_value_dict) will be added in v2.0), this probably makes it unnecessary to use multiple connections

Can you do that from scratch too somehow or is it just a strange bug?

TimMcCool commented 1 month ago

Actually there's no limit for how many cloud variables can be set simultaneously, but you should wait 06667 * n seconds between cloud variable sets (when setting n cloud variables simultaneously) to avoid getting ratelimited. I found this out experimentally.

If you wait shorter, you'll be able to set cloud variables much faster for a few seconds (I was able to perform 32 cloud variable sets in a second), but then you'll be ratelimited for another few seconds, and while you're ratelimited your cloud variable sets are ignored by the server.

You can't do it from Scratch

TimMcCool commented 1 month ago

I just did some more tests and it looks like when setting cloud variables while waiting less than 0.1 * n between sets, it only works for about 2 - 3 minutes, and then you'll still be ratelimited for a few seconds.

I'll make scratchattach automatically increase the wait time after a while to prevent this.

TheCommCraft commented 1 month ago

Maybe also add type annotations

TheCommCraft commented 1 month ago

I feel like using threading.Event would be more appropriate than threading.Condition

TimMcCool commented 1 month ago

True. I'll use threading.Event instead

TimMcCool commented 1 month ago

scratchattach 2.0 will be released on Friday (10/11/24)

TheCommCraft commented 1 month ago

That date has already passed

Tim @.***> schrieb am So., 6. Okt. 2024, 06:44:

scratchattach 2.0 will be released on Friday (10/06/24)

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2395292083, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCKMZWLATYE7ZO5LPXJ75TZ2C53PAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGI4TEMBYGM . You are receiving this because you commented.Message ID: @.***>

TimMcCool commented 1 month ago

scratchattach 2.0 is complete now. I decided to cancel asyncio cloud requests because I didn't have time for implementing them (and now that threading.Event has been implemented, they're already much less resource intensive). I'll maybe add them later.

If you want to contribute to the library by implementing these yourself, you're free to do so by submitting a pull request to the v2 branch (I suggest creating a new CloudRequestsAsync class that inherits from CloudRequests and overrides methods where it is needed to implement asyncio. Doesn't have to be bug-free).

Tomorrow I'll update the docs and release v2.0

SpyC0der77 commented 1 month ago

Update???

TimMcCool commented 1 month ago

sorry, my university semester started and now i have very little time to work on this. I'll try to finish the docs this weekend

SpyC0der77 commented 1 month ago

Good luck with university!

On Thu, Oct 17, 2024 at 2:27 AM Tim @.***> wrote:

sorry, my university semester started and now i have very little time to work on this. I'll try to finish the docs this weekend

— Reply to this email directly, view it on GitHub https://github.com/TimMcCool/scratchattach/issues/242#issuecomment-2418633746, or unsubscribe https://github.com/notifications/unsubscribe-auth/BJAV6RFZEC5UV46Q74GJS53Z35KFLAVCNFSM6AAAAABO2X47SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGYZTGNZUGY . You are receiving this because you commented.Message ID: @.***>

TimMcCool commented 4 weeks ago

scratchattach 2.0.0 is released now! Use pip install -U scratchattach to update.

Good luck with university! …

Thanks!

TheCommCraft commented 3 weeks ago

What are you studying?

TimMcCool commented 3 weeks ago

What are you studying?

computer science

TimMcCool / scratchattach

Ideas for scratchattach 2.0 #242