Expire only boosts - Githubissues

lapineige commented 5 years ago

Hello,

Is it possible to add an option to expire only our boosted toots, instead of expiring all statuses ?

Thanks

kensanata commented 5 years ago

Do you want to add such a feature? I don’t think I will be adding this myself, though. Perhaps somebody else will volunteer.

BartG95 commented 5 years ago

@lapineige what exactly do you mean by boosted toots? Do you mean: a) Your statuses, that are boosted by others (i.e. you only want to retain your statuses are not boosted); b) Statuses from others, that you boosted (i.e. you want your account to show only your toots)

Option A seems difficult to implement, but option B should be relatively easy. Essentially, you need a version with these lines removed: https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L44-L45 For the time being, you can make a clone of the repo, remove these lines and install the package locally with pip install -e . (replace the dot with the path to your local repo, or just cd into the repo.)

BartG95 commented 5 years ago

If you want to make a patch, you will need to add an option here: https://github.com/kensanata/mastodon-backup/blob/43deaa2a53e14c6e4f2d653fe71647d61c33d87f/mastodon_archive/__init__.py#L153 Then, modify the delete function to take an extra boolean here: https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L36 Then, transform this else-statement into an elif-statement: https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L44 Then, modify the call to the delete function here: https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L108 Feel free to write a patch ;)

lapineige commented 5 years ago

@lapineige what exactly do you mean by boosted toots? Do you mean: a) Your statuses, that are boosted by others (i.e. you only want to retain your statuses are not boosted); b) Statuses from others, that you boosted (i.e. you want your account to show only your toots)

I mean option b :)

I don't have the time right now, but I'll have a look. Thanks for your help, I appreciate it :)

LeonardoFurtado commented 5 years ago

i need it, maybe i'll implement.

kensanata commented 5 years ago

Very cool, thanks!

lapineige commented 5 years ago

Hello, any update on this ? :)

kensanata commented 5 years ago

I’m not working on it, if that’s what you mean… Sorry!

lapineige commented 4 years ago

I'm actually wondering if it expires boost right now… I've searched my old post, and the classic toots seems to be deleted, while boosts are still there. @kensanata does Mastodon-backup expire boosts ?

kensanata commented 4 years ago

I think it does! But perhaps there’s a bug?

lapineige commented 4 years ago

How can I check this ?

lapineige commented 4 years ago

Another person reported to me that her boost were not expired. 2 questions:

can it expire boost that has not been archived ?
how can I check if the boosts are taken into account when I expire my statuses ?

kensanata commented 4 years ago

What happens when you run mastodon-archive expire --older-than 8 --collection favourites --confirm lapineige@your.instance, for example? What sort of output do you see?

lapineige commented 4 years ago

It's just expiring toots… normal output, Loading archive then expiring.

Note that there is an issue when expiring favorites:

without --pace:

Traceback (most recent call last): File "/home/name/.local/bin/mastodon-archive", line 10, in sys.exit(main()) File "/home/name/.local/lib/python3.7/site-packages/mastodon_archive/init.py", line 293, in main args.command(args) File "/home/name/.local/lib/python3.7/site-packages/mastodon_archive/expire.py", line 104, in expire mastodon.ratelimit_method = 'wait' UnboundLocalError: local variable 'mastodon' referenced before assignment
with --pace, it doesn't take it into account and goes full speed.

kensanata commented 4 years ago

I’m interested in seeing whether you specify the collection – if it’s expiring toots it is not expecting favorites and vice verse.

Mellerin commented 4 years ago

Hi! Looks like i'm the other one who have troubles with the same issue. I ran mastodon-backup with several options, it built my archive, deleted my toots, my favorites, downloaded my media files. All i can see in my profile now are still the toots from other people that i boosted.

mastodon-archive report --all --include-boosts me@myinstance.com
Considering the entire archive
Statuses:             21909
Boosts:                3296
Media:                  556

Top 10 hashtags:
#pouetradio(276) #mastodon(97) #tonfilmenpetit(80) #photo(65)
#photography(54) #tootradio(41) #photographie(40) #nsfw(40)
#mercredifiction(39) #question(35)

Favourites:           10797
Boosts:                   0
Media:                 2048

No more statuses on my account (when connected on the web ui of my instance), but only the 3K+ boosts reported here. Maybe I just don't run the appropriate command? I test the one you suggested earlier mastodon-archive expire --older-than 4 --collection favourites --confirm me@myinstance.com but nothing new, even if it ran smoothly, even if it expired 10197 something but I don't know what are those 10197 since it doesn't match any numbers reported above (i guess it's some of the favourites, but not all)/ I also tried mastodon-archive expire --older than 4 with and without all the --collection but it just now says "no statuses/favourites/mentions are older than x weeks".

If I understand it clearly, your program doesn't erase all my stuff with the same command, need to run it with appropriate command (not a bad thing). So : 1- build an archive mastodon-archive archive --with-mentions myaccount 2- delete regular statuses (toots) : mastodon-archive expire --older-than 8 --collection statuses --confirm myaccount 3- delete favorited : mastodon-archive expire --older-than 8 --collection favourites --confirm myaccount 4- delete notifications : mastodon-archive expire --older-than 8 --collection mentions --delete-other-notifications --confirm myaccount

Basically all that you kindly suggest in your example setup : https://github.com/kensanata/mastodon-backup#example-setup

extra-step : 5- mastodon-archive expire --older-than 8 --confirmed myaccount (still expires some stuff)

6- Still have all the boosts! ^_^! (statuses from other people that i boosted)

Sorry, I'm not a programmer and I just don't understand what BartG95 suggested to do earlier :/ https://github.com/kensanata/mastodon-backup/issues/40#issuecomment-451663947 I tried some modifications without any results but maybe i didn't ran it the good way (not a dev!)

Besides that, thanks for your program and your help, have a good day ^_^

lapineige commented 4 years ago

I’m interested in seeing whether you specify the collection – if it’s expiring toots it is not expecting favorites and vice verse.

Well I specify --collection favourites

kensanata commented 4 years ago

Hm. Let's see… This is the code that does the deleting. Apparently you need to specify the statuses collection in order to "unreblog" (unboost) a status.

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["id"])
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

So, either the status isn't marked as "reblog" on the archive, or the code isn't finding it, or the status isn't in the archive in the first place.

So, looking for a status in the JSON file with "reblogged": false...

I'm opening my text editor, loading the JSON file, and looking for this string…

Finding this one:

    {
      "id": 100148697908936006,
      "created_at": "2018-06-04T21:30:56.281000+00:00",
...
      "url": "https://octodon.social/@kensanata/100148697908936006",
...
      "reblogged": false,
...
      "reblog": {
        "id": 100148655205233270,
        "created_at": "2018-06-04T21:19:55+00:00",
...
        "url": "https://trunk.mad-scientist.club/@algernon/100148654631327735",
...
        "reblogged": true,
...

Let's check! I'm searching for https://trunk.mad-scientist.club/@algernon/100148654631327735 in my Mastodon session, and finding it. I'm checking the "boost" icon and it's off, so I haven't boosted it.

So, it seems to be working.

I think I'd need this kind of investigation from one of you.

For example:

visit your profile and scroll down until you find a boosted toot that should have been unboosted
right-click on the timestamp and "Copy Link Location" (or whatever your browser says)
open your JSON archive in a text editor
search for the URL you copied
did you find it?
as you can see in the example above, your status status (in the example above the one with id 100148697908936006) contains the boosted status in the reblog section (in the example above the one with id 100148655205233270), is this reblog status marked with "reblogged": true?
when you visit the URL of the boosted status, does it look boosted?
what version of Mastodon is your instance running? (see the very bottom of the /about/more link, e.g. I'm checking https://octodon.social/about/more)
what version of Mastodon.py are you running? (something like locate "Mastodon.py"|grep dist-info\$ might help?)
what version of Mastodon Archive are you running?

Mellerin commented 4 years ago

Hello and thanks for your reply ^_^ Firstly, versions : mastodon-archive v1.3.0 mastodon.py v1.5.0 (the one that was installed while installing mastodon-archive) mastodon instance v3.0.1 (mamot.fr) So I visited my profile, scroll down and find many that should have been unboosted, like these ones (it's an image which should be deleted in 180 days) Regarding the date I'm sure i ran all mastodon-backup commands multiples times on these dates, so they shouldn't appear as boosted here but as you can see in the picture, they are still boosted. I take the one from Unpied copy its link and search for it in my archive :

    {
      "id": 102999751926001273,
      "created_at": "2019-10-21T09:51:28.130000+00:00",
...
      "language": null,
      "uri": "https://mamot.fr/users/EllerinPrv/statuses/102999751926001273/activity",
      "url": "https://mamot.fr/users/EllerinPrv/statuses/102999751926001273/activity",
      "replies_count": 0,
...
      "reblogged": false,
...
      "reblog": {
        "id": 102999749014241944,
        "created_at": "2019-10-21T09:50:32+00:00",
...
        "uri": "https://birdsite.link/users/Unpied/statuses/102999748288513511",
        "url": "https://birdsite.link/@Unpied/102999748288513511",
        "replies_count": 0,
        "reblogs_count": 5,
        "favourites_count": 2,
        "favourited": false,
        "reblogged": true,
...

So even if i visit this particular toot alone, it appears as boosted. If i understand the reblog section says i boosted it but the first "reblogged": false before that says it is not boosted (un-boosted) anymore right now? So, maybe mastodon-archive seems to work as expected but maybe there's something odd when sending unboost/unreblog command towards the instance which doesn't execute the unreblog command or doesn't receive it? Last time i ran the expire command i didn't see any errors in the output. Is there a way to have a more detailed report from the program in order to see if for example the mastodon instance ignore the unreblog command? Anyway, again, thanks for your time :bowing_woman:

kensanata commented 4 years ago

If i understand the reblog section says i boosted it but the first "reblogged": false before that says it is not boosted (un-boosted) anymore right now?

I'm not sure that's how you read it. I read it like this: you did something, so there's a status associated with it. The thing you did was boost (reblog) another status (which is now marked as reblogged true). That is, the outer status (your status) is never marked as reblogged. The status that you're boosting (the one that's is part of the the reblog attribute) is always marked as reblogged true). In short, the data I'm seeing says: you reblogged this status.

maybe mastodon-archive seems to work as expected but maybe there's something odd when sending unboost/unreblog command towards the instance which doesn't execute the unreblog command or doesn't receive it?

That is of course always possible but not a very satisfying answer. I wonder what's going on.

Last time i ran the expire command i didn't see any errors in the output. Is there a way to have a more detailed report from the program in order to see if for example the mastodon instance ignore the unreblog command? Anyway, again, thanks for your time 🙇‍♀

The code also says the following:

    def matches(status):
        created = datetime.strptime(status["created_at"][0:10], "%Y-%m-%d")
        deleted = "deleted" in status and status["deleted"] == True
        pinned = "pinned" in status and status["pinned"] == True
        return created < cutoff and not deleted and not pinned

    statuses = list(filter(matches, data[collection]))

That is, we're not doing anything to statuses that are marked as deleted. So I went back to my JSON file and checked status 100148697908936006 and indeed, at the very end:

    {
      "id": 100148697908936006,
...
      "reblog": {
        "id": 100148655205233270,
...
        "reblogged": true,
...
        "account": {
          "id": 23975,
          "username": "algernon",
...
        },
...
      },
      "application": null,
      "account": {
        "id": 12540,
        "username": "kensanata",
...
      "deleted": true
    },

That is, my own "reblogging action" (id 100148697908936006) is marked as deleted.

I wonder, however. Perhaps my code is confusing the two ids? What happens is that it goes through the list of statuses, finds my "reblogging action" (id 100148697908936006) and tries to delete it.

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["id"])
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

As you can see, the delete code then checks whether the status contains a "reblog" and if it does, it unreblogs the id of the status – but not the id of the reblogged status. That is, 100148697908936006 is unreblogged, not 100148655205233270.

So now I'm wondering: perhaps the code should be changed to the following?

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["reblog"]["id"]) # ← THIS!
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

Can you make this change to your copy and give it a try?

(Sadly, I will be leaving for a trip, soon, and won't be able to get back to this for a few weeks.)

Mellerin commented 4 years ago

Hello, Ok, i admit all these kind of statuses and their mechanics are a bit confusing me ^_^! So, in fact if I boost someone else's toot, it creates my own toot status id which is marked as a reblog but not reblogged, then the reblog section refer to the original status (toot) which is now marked as reblogged=true. Well, sadly, at the very end of this (my) status id, I also find a "deleted": true (just before the begining of another { "id" : ... I will check your modification and let you know. I messed with the little Raspberrry Pi which drives your program so i need to re-install it before. From my side this boost/reblog is not a hurry, so please enjoy you trip (well, as fas as it's a trip you can enjoy). Thank you o/

kensanata commented 4 years ago

I'm back from my trip. Did it work as intended, @Mellerin?

Mellerin commented 4 years ago

Hello and wb o/ I hope everything's going well for you during these troubled times.

Well, it worked i suppose, it just lets some boosted toots and the very end of my own timeline. I didn't investigate yet exactly why except that the (few) statuses that remain go to a "the page you are looking for isn't here" on the mastodon web interface, maybe deleted statuses but looks like it's not a problem with mastodon-archive ^_^ So, back to mastodo-archive, what i did :

create a new venv : python3 -m venv mastobackup-dev
activate and go in it : source mastobackup-dev/bin/activate
git clone mastodon-archive in it : (mastobackup-dev) me@host:~ $ git clone https://github.com/kensanata/mastodon-backup.git

edit the file : ./mastodon-backup/mastodon-archive/expire.py and change the code as stated previously :

if collection == 'statuses':
    if status["reblog"]:
        mastodon.status_unreblog(status["reblog"]["id"])

now installing the modified local repo : $ pip3 install -e ./mastodon-backup
create a new dir for archive and go in : mkdir tootback and cd ./tootback
archive new archive : ~/tootback $ mastodon-archive archive myaccount@mamot.fr
token validation
delete statuses : ~/tootback $ mastodon-archive expire --older-than 3 --confirm myaccount@mamot.fr
token validation again (maybe i missed something here for it to ask again, the *secret files exist after that)
expiring ## / 3684
OK \o/

I detailed all the things i've done, not sure all the steps are necessary neither the right ones to do ^_^! So it looks like the little code modification does the job. Thank you :+1:

Little edit : btw I'm not sure it answers exactly what @lapineige asked at the very beginning ? Done a PR, please ignore it if you want ^_^

kensanata commented 4 years ago

If this is good enough for you, @lapineige, please close this issue. Otherwise I'm going to assume that you still want an option to expire only boosts without expiring your own toots.

lapineige commented 4 years ago

Sorry I read the whole conversation but had no time to react/test it myself.

If I understand well the changes, now I need to make my archive again (to create the "reblog" category) and then remove toots just like before and it will include all reblogs ?

Well an option would be cool, but maybe I can make the PR myself :)

kensanata commented 4 years ago

No change is required for you; from now on it should simply unboost the correct id because it uses the id of the boosted status to unboost instead of the irrelevant id of your boost action.

lapineige commented 4 years ago

Ok thanks. I guess I need to wait for the next release ?

kensanata commented 4 years ago

Depends on how comfortable you are with simply installing from source. 😀

I just use the advice given here: https://github.com/kensanata/mastodon-backup/blob/master/README.md#development Check it out using git; run pip3 install --upgrade --editable .

lapineige commented 4 years ago

Finally I don't think I'll try a non-release version. Do you have an idea about when the next release can be done ?

kensanata commented 4 years ago

I made a release 1.3.1.

lapineige commented 4 years ago

Yeah 🎉 Thanks a lot !

So I tried it… I was planning to delete old boosts, on data that was already "deleted" (but not the boosts). But it says that no status is older than [X] weeks…

Could it be that it tried to delete them, marked them as "deleted", and then it won't try again because it is supposed to already be removed ?

kensanata commented 4 years ago

Hm, could be. What do you think we should do, now? Perhaps we could write a command that goes through all the boosts and marks them as not deleted?

lapineige commented 4 years ago

That's a possibility indeed 🤔 @Mellerin do you think you could (and want to) do it, or should I try to ?

kensanata commented 4 years ago

I am super confused when I read through this thread. 😅

I wrote a new subcommand, fix-boosts which does what I proposed. I run it as follows:

mastodon-archive fix-boosts kensanata@octodon.social

If you don't confirm, it prints how many changes it would have made, and twenty examples. To verify, you need to paste these URLs into the search box of your Mastodon account so that you can see whether they are still boosted. This is important: if you just click on the links and look at them in your browser, they will always look unboosted.

Let me know if that fixes it for your?

I would like to avoid making a release for this. I'd love for you to test it from the git repository:

git clone https://github.com/kensanata/mastodon-backup.git
cd mastodon-backup
pip3 install --upgrade .

That should get you commit bc891f7.

Once you do confirm (make a copy of your archive first, just in case!) it makes the changes and then the next time you run mastodon-archive expire it should try to expire all those boosts once again.

lapineige commented 2 years ago

Coming back to this issue after a while… I am a bit lost :sweat_smile:

I will give it a try :slightly_smiling_face:. Could you please merge current commits to this version ? I don't know if that necessary to actually try it, but I would prefer to use the most up-to-date version just in case in creates any compatibility issue or such :sweat_smile:

kensanata commented 2 years ago

This has been integrated into mastodon-archive for a long time time, now. Two years ago was the last commit referring to fix-boosts.

lapineige commented 2 years ago

Ok sorry I didn't catch that change. Thank you :)

lapineige commented 2 years ago

So the fix-boosts command does give a list of still boosted toots. Great !

But how can I expire them ? And what if I want to expire only boosts older than a specific date ?

Thank you

lapineige commented 2 years ago

Ok I think I figured out : I need to run the fix-boosts command (with --confirmed option) to mark them as non expired, then run the expire command.

But that means I can't expire only boosts. Would that be a possible addition ? As the command can list boosts only, I guess it would be possible to reuse that list in the expire command… I guess it would be a new --collection boosts argument ? Ready that code https://github.com/kensanata/mastodon-backup/blob/main/mastodon_archive/fix.py#L39= I understand that this new option code would "just" need a condition like this if status["reblog"] : [command to expire it]. Am I right ?

kensanata commented 2 years ago

Hm, indeed, there currently is no option to delete just boosts. I think you are right. In expire.py lines 38–54:

def delete(mastodon, collection, status):
    """
    Delete toot or unfavour favourite and mark it as deleted. The
    "record not found" error is handled elsewhere. Mentions cannot be
    dismissed because mastodon.notifications_dismiss requires a
    notification id, not a status id.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["reblog"]["id"])
            status["deleted"] = True
        else:
            mastodon.status_delete(status["id"])
            status["deleted"] = True
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
        status["deleted"] = True
    elif collection == 'boosts' and status["reblog"]:
        mastodon.status_unreblog(status["reblog"]["id"])
        status["deleted"] = True

This assumes we don't want to change the existing functionality for the "statuses" collection. I'm not sure what would be best, here.

For the command line option in __init__.py, lines 186–189:

    parser_content.add_argument("--collection", dest='collection',
                                choices=['statuses', 'favourites', 'mentions', 'boosts'],
                                default='statuses',
                                help='delete statuses, unfavour favourites, clear mention notifications, or only boosts')

Now there's still a problem: boosts are stored as statuses "containing" the boosted status in the "reblog" attribute. If we use "--collection=boosts" then the code in expire.py will try to load the collection which doesn't exist. It's actually the "statuses" collection we need to load. Yuck.

Something needs to change in expire.py line 95:

        statuses = list(filter(matches, data[if collection == "boosts" then "statuses" else collection]))

All of the code examples untested. 😢

I'm leaving on a trip for the next few days so there's going to be no answer for a while. Give it a try, perhaps with a test account. 😅

We could of course just add a different option, --boosts-only, which only works for --collection=statuses. 🤔

lapineige commented 2 years ago

All of the code examples untested. cry

I'm leaving on a trip for the next few days so there's going to be no answer for a while. Give it a try, perhaps with a test account. sweat_smile

Wow that's great ! Thanks a lot for your help !

I think I'll try some dry-run first :D

kensanata / mastodon-archive

Expire only boosts #40