Temporary containers not removed with new sync feature

neekt commented 4 years ago

Temporary Containers Version: 1.7
Firefox Version: 73.0

I've just noticed that with Firefox's new Multi-Account Containers sync feature, temporary containers are being synced across browsers on different machines, and then not removed, even when all instances of the container (e.g., tmp4) are closed in both browsers. Completely closing both browsers and then re-opening doesn't get rid of them either. They can be manually removed, if you manually remove them from both browsers and then sync Firefox, but in my case I have found dozens of temporary containers are being synced and kept, so this is not practical.

I assume it's possible to deactivate the container syncing feature (it asks whether to activate it upon update to the new Multi-Account Containers version), but I actually find it quite useful. So if there is some way to prevent temporary containers from being synced (ideally), or (less ideally) to allow them to still be automatically removed even after they've synced, that would be great.

This is using Automatic Mode, deleting no longer needed temporary containers after the last tab in it closes, and reusing available numbers in settings. Everything else is set to default.

stoically commented 4 years ago

That's unfortunate. I've mentioned this during review of the sync feature, and it was said that it should work. I'll look into it at some point, but it might be worth filing this in the MAC issues as well, since in theory the containers should get deleted eventually.

CartCaved commented 4 years ago

Maybe it's because @neekt is reusing available numbers, and the dedupe gets screwed up, as per your exchange here https://github.com/mozilla/multi-account-containers/pull/1611#discussion_r366477898 and the list of deleted containers gets confused?

ChrisWiegman commented 4 years ago

I'm hitting this myself but turning off sync hasn't helped. I have 1,000s of containers and manually deleting (which is not fun) only works for a minute or two until they all come back.

stoically commented 4 years ago

If turning off sync in MAC lets the container reappear, then that's definitely a MAC bug that you might want to report there.

Did you have "Reuse available numbers" active in TC while MACs sync was active?

okradonkey commented 4 years ago

Just a heads-up; I have the same issue of TCs not getting deleted, but I'm not using the MAC extension at all - I'm using Containerise - so the issue might not be in MAC itself, but rather elsewhere in Firefox or the Sync system. I only use one computer but I do have Sync enabled. TC is set to keep counting.

Edit - Sorry, false alarm. I did have about 20 TCs lingering in my list (which is why I was here to begin with) but they may have been leftover from when I was trying out the updated MAC. I had MAC with its sync enabled for a few days before deciding to disable it and continue with Containerise.

Now (without MAC), after deleting the leftover TCs and trying to reproduce the issue, I've found that everything is working perfectly; newly abandoned TCs are deleted immediately and don't reappear. Thanks for your help.

stoically commented 4 years ago

@okradonkey TCs not getting deleted at all without MAC is a different issue (since the container sync is part of MAC), and definitely shouldn't happen. Would you mind:

Setting the removal delay to instant (Preferences > "Delete no longer needed Temporary Containers" > "After the last tab closes")
Preparing an debug log
Opening and closing an TC
Posting the debug log as new issue?

dveditz commented 4 years ago

sh4r1k7 commented 4 years ago

Can anyone suggest a way to delete all the containers? I have over a thousand, it's inhumane to delete them by hand. I turned off sync but they're still there slowing everything down.

stoically commented 4 years ago

@3dc1d3

You could remove all containers that aren't currently active temporary containers, though that will remove all your permanent containers that you might want to keep.

To remove all containers except currently active temporary containers:

Backup your profile(s)
Navigate to about:debugging#/runtime/this-firefox
Click "Inspect" right to Temporary Containers
Click "Console"
Insert into the console Edit: Might want to use https://github.com/stoically/temporary-containers/issues/371#issuecomment-617626349 instead
If you're getting a "Scam warning", type "allow pasting" into the console, remove it again, and then insert code from last step again
Press Enter and wait for the console to print done

stoically commented 4 years ago

Vinnl commented 4 years ago

Thanks @stoically. Unfortunately I just used that to remove more containers than I wanted due to an unfortunate press of the Enter key, but for those who want to avoid that, this snippet should only remove containers starting with tmp:

const containers = await browser.contextualIdentities.query({});
await Promise.all(containers.map(container => {
  if (tmp.container.isTemporary(container.cookieStoreId)) return;

  const prefix = 'tmp';
  if (container.name.substring(0, prefix.length) !== prefix) {
    return;
  }

  // could add exceptions here for other permanent containers,
  // e.g. by looking into the containers.json, like so:
  // if (container.cookieStoreId === "firefox-container-123") return;

  return browser.contextualIdentities.remove(container.cookieStoreId);
}));
console.log("done");

(People using a different prefix can replace tmp in const prefix = 'tmp'; with their prefix.)

sh4r1k7 commented 4 years ago

thanks @stoically and @Vinnl - finally got around to it. 👍

serhumanos commented 4 years ago

hi, i try the @Vinnl code, it removes all tmp# containers.. but after a minute, all of them reappear. i disabled firefox sync for addons, and MAC sync too, and tmp# (and other containers not "tmp#" but deleted on the past, are keep reappearing) my firefox is too slow, is there any other alternative?

crssi commented 4 years ago

Would you care to uninstall MAC and try again?

serhumanos commented 4 years ago

Would you care to uninstall MAC and try again?

i broke my firefox profile doing the last operation. i load an old backup of my profile, then deactivate sync for my firefox account (for now i prefered to leave manual) and the problem is not present now (i know this is not a solution) i had to block the access to profile/wave/logs, because the directory continous creating a lot of log files that collapse my disk space.

i am wating for a real solution for this, if you want i can try do something else (after re-backup my profile again xD)

thanks!

abulgatz commented 4 years ago

Anyone have any luck with any workarounds?

krishnanunnir commented 4 years ago

I am also facing this issue, what is worse is due to some bug, 100 temporary containers were created and they are not getting deleted.

svendsvin commented 4 years ago

Same here. Script does not work. The massive amount of tmp containers really slows down FF.

krishnanunnir commented 4 years ago

@svendsvin You can delete the containers in a much easier way by going to Container Settings in Preferences.

avian2 commented 4 years ago

To spare anyone else the time it took me to find the menu @krishnanunnir is talking about. It's not in the add-on preferences. It's in the main Firefox preferences, under the "General" tab.

kaushikgopal commented 4 years ago

actually the quickest way is to just type "about:preferences#containers" in the url bar. I have it bookmarked just for this purpose :)

rob-miller commented 4 years ago

"about:preferences#containers" is really helpful, would still be very useful to have a button in the Temporary Containers controls somewhere to 'delete all temporary containers now'. If there's a disagreement with MAC sync at some point it seems like I could just shutdown browsers and clear them individually using this button.

darthvader666uk commented 4 years ago

actually the quickest way is to just type "about:preferences#containers" in the url bar. I have it bookmarked just for this purpose :)

thank you!

svendsvin commented 3 years ago

After updating to 82.02 the problem seems to have disappeared. Sync is enabled. I restarted FF several times and all temporary containers are removed.

serhumanos commented 3 years ago

After updating to 82.02 the problem seems to have disappeared. Sync is enabled. I restarted FF several times and all temporary containers are removed.

same here! thanks!

rtrind commented 3 years ago

Not for me. As long as a second computer is used, all my temporary containers from the past come back. I tried several orders of enabling/disabling the sync feature on the addon and enabling/disabling the addon syncing from the cloud. They always come back for me. 82.02 was no better than before for me.

serhumanos commented 3 years ago

the first thing i did, was enable addons sync and enable sync from containers.. after that the hundred of tmp containers reappears :( next, i started to delete each of tmp containers (included non tmp or old containers backedup) after delete all of them, they are gone forever.. i lost some websites asigned to containers that mixed current and backedup, but after that, all is fine now. i hope this can help.

rtrind commented 3 years ago

I did the same thing. But when I went to my second machine to do the same process, the temporary containers were back on the primary machine again. After cleaning them up, they were back on the second one, and so on...

serhumanos commented 3 years ago

i understand.. sorry for my next suggestion i know that not resolves the problem directly but... if you copy your firefox profile directory to the second machine (after your "clean" your first profile) ? maybe it can work

akhyarrh commented 3 years ago

My current workaround (note: will delete all container and reset to default). The pros of this step is dont need new profile. I used TC with MAC & FBC & Google Container. YMMV.

disable sync in MAC
click Disable Extension until can uncheck Enable Container Tabs (see image below)

restart Firefox
enable all container add-ons again (about:addons)

alexhk commented 3 years ago

Thanks @akhyarrh

Seems I finally came to the right place. I can't get rid of my old temporary containers, no matter what I try.

I am willing to go the reset delete all container and reset to default route, does anybody have a recommendation how to comfortably recreate/edit all my good and useful containers after a reset? Nice and clean export/edit/import would be too much to ask for I guess ;)

timonoj commented 3 years ago

Just to add, this is not solved. The script didn't fix it.


const containers = await browser.contextualIdentities.query({});
await Promise.all(containers.map(container => {
  if (tmp.container.isTemporary(container.cookieStoreId)) return;

  const prefix = 'tmp';
  if (container.name.substring(0, prefix.length) !== prefix) {
    return;
  }

  // could add exceptions here for other permanent containers,
  // e.g. by looking into the containers.json, like so:
  // if (container.cookieStoreId === "firefox-container-123") return;

  return browser.contextualIdentities.remove(container.cookieStoreId);
}));
console.log("done");
Uncaught (in promise) ReferenceError: tmp is not defined
    <anonymous> debugger eval code:6
    <anonymous> debugger eval code:5
    async* debugger eval code:19

This is the error I get. I also tried manually editing the containers.json. It makes the containers disappear, but not for long. Whenever I install fresh the MAC addon on any firefox, the moment I enable sync, it re-downloads thousands of tmp containers. The messed data is in ithe sync cloud. And after messing this up, it never syncs again. Fixing the data locally doesn't sync back. Any new firefox install will still download thousands of tmp containers.

nnnnicholas commented 3 years ago

Reposting two useful comments from a Hacker News discussion:

jannes said:

You can reset your sync data for any extension in Firefox. Just go to about:debugging, click on "This Firefox", locate the extension and click the "Inspect" button. In the case of MAC this should lead you to about:devtools-toolbox?type=extension&id=%40testpilot-containers

In the devtools that open you need to go to the Console tab and run this command: await browser.storage.sync.clear()

If you want to see the data before you delete it, you can show it with this command: await browser.storage.sync.get()

krastanov said:

I have the same problem (which supposedly does not happen to all users of that extension). I fix it by running this script once a month (it deletes the temporary containers from the containers.json file):
import json

with open('containers.json.bac') as f:
    s = f.read()
    j = json.loads(s)
    ids = [i for i in j['identities'] if not i['name'].startswith('tmp')]
    print(len(j['identities']), len(ids))
    j['identities'] = ids
    with open('containers.json', 'w') as wf:
        wf.write(json.dumps(j))

avian2 commented 3 years ago

The await browser.storage.sync.clear() only clears local data. It does not help when the local data does not sync with the remote end due to the HTTP 507 Insufficient Storage error. The server refuses to delete anything on its end after the quota has been reached. It will, however, happily restore hundreds of containers back to you on a fresh MAC install.

I just tried the await browser.storage.sync.clear() recipe again and got the same result as already reported in https://github.com/mozilla/multi-account-containers/issues/1675#issuecomment-619615064

EDIT: I did some more digging and it seems also manually clearing data on the remote end doesn't help. See my comment here: https://github.com/mozilla/multi-account-containers/issues/1675#issuecomment-768186221

IssueHuntBot commented 3 years ago

@stoically has funded $50.00 to this issue.

Submit pull request via IssueHunt to receive this reward.
Want to contribute? Chip in to this issue via IssueHunt.
Checkout the IssueHunt Issue Explorer to see more funded issues.
Need help from developers? Add your repository on IssueHunt to raise funds.

stoically commented 3 years ago

Regarding the IssueHunt bounty: This would need to get fixed in MAC. I can however review and potentially merge a PR over there. A simple thing that should already do the trick could be if MAC would allow specifying a regex pattern to ignore containers with certain names, like tmp*, when syncing

avian2 commented 3 years ago

There are many opened MAC issues that report enabling sync leads to thousands of containers and/or hitting the storage quota on the sync server. Some of these specifically report not using the Temporary Containers extension and ending up with many copies of just the default MAC containers. e.g. https://github.com/mozilla/multi-account-containers/issues/1912, https://github.com/mozilla/multi-account-containers/issues/1885, https://github.com/mozilla/multi-account-containers/issues/1839, https://github.com/mozilla/multi-account-containers/issues/1799, https://github.com/mozilla/multi-account-containers/issues/1737, https://github.com/mozilla/multi-account-containers/issues/1732 https://github.com/mozilla/multi-account-containers/issues/1661

This makes me think that just ignoring tmp* containers won't help. It also won't help people like me that already have a hosed profile and/or sync account that we can't seem to recover.

I spent hours looking at this and I still don't understand what's going on. There are pages upon pages of docs on Mozilla's sync infrastructure out there. I found it very hard to figure out what applies and what not in this case. A lot of documentation seems to be outdated. Different extensions seem to use a completely different API for it.

Two things that I found suspicious and couldn't figure out, in case someone else decides to look at it. I'm pretty much done trying to fix this:

There are references to a "backup" in MAC code (https://github.com/mozilla/multi-account-containers/blob/master/src/js/background/sync.js#L111) I don't understand how this works and it might be the reason why deleted containers keep coming back.

During sync, Firefox makes HTTP requests to two separate services: sync-xxx.sync.services.mozilla.com and webextensions.settings.services.mozilla.com. In my experiments I've manually cleaned out the webextensions one, but it might be that something remains on the other one. I couldn't figure out what's the role of the other one and whether MAC stores any data there.

stoically commented 3 years ago

@avian2 Thanks for digging into it and sharing your findings

This makes me think that just ignoring tmp* containers won't help

It could at least help with new accounts / users to not needlessly trying to sync temporary containers - but not with other bugs in MAC or sync in general, right.

Modged commented 3 years ago

Hi how to delete all my tmp containers (+8000) on windows ? I thought the sync feature was a good idea for the manual created containers with their websites linked to it... But it doesnt work am I right ?

Thanks by advance :)

timonoj commented 3 years ago

Check Containers Helper. It's ANOTHER addon...It lets you easily tag multiple containers (like, all containers starting with 'tmp') and delete/clone....anything. And then, try to sync again.

On Thu, 15 Apr 2021 at 02:57, Modged @.***> wrote:

Hi how to delete all my tmp containers (+8000) on windows ? I thought the sync feature was a good idea for the manual created containers with their websites linked to it... But it doesnt work am I right ?

Thanks by advance :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/stoically/temporary-containers/issues/371#issuecomment-819757245, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPXB55FG67B6TAVIXDJE6LTIXQTJANCNFSM4KY4O6QA .

FlorianWendelborn commented 3 years ago

@Modged https://github.com/mozilla/multi-account-containers/issues/1885#issuecomment-808897292

Modged commented 3 years ago

@FlorianWendelborn Thanks I have seen that but I have no clue how to use it any help would be welcome Edit : even if I have found a solution I would like to know how to use it ?

@timonoj I am going to check thanks :) Edit : Worked !! (How can I resync I don't find the option like in the screens captures as above)

sshquack commented 2 years ago

Disabling sync in MAC seems to have fixed my issue. YMMV

Here is a video of all my temporary containers being deleted after last tab in it closes.

https://user-images.githubusercontent.com/5678459/135193185-918e8750-0be5-473d-aa02-c723dd592e83.mov

Multi-Account Containers Version: 7.4.0
Temporary Containers Version: 1.9.2
Operating System + Version: macOS Catalina 10.15.7 (19H1417)

worldofgeese commented 2 years ago

I'd really like to keep syncing and not get swamped by new tmp containers every time I login to a new session.

stoically commented 2 years ago

@worldofgeese Seeing that you fiddled with JS, if you'd like to help, here's how: https://github.com/stoically/temporary-containers/issues/371#issuecomment-776469028

IssueHuntBot commented 2 years ago

@stoically has cancelled funding for this issue.(Cancelled amount: $50.00) See it on IssueHunt

stoically commented 2 years ago

Opened a PR over in MAC: https://github.com/mozilla/multi-account-containers/pull/2231

onebod commented 2 years ago

How do I simply delete all these TCs without doing it one-by-one via MAC?

Valentin-N commented 2 years ago

I've used Containers Helper in the past to mass delete containers.

neekt commented 2 years ago

I've used Containers Helper in the past to mass delete containers.

This extension was helpful for mass deleting all the tmp containers that kept syncing - thanks! - but no containers are syncing at all now that that mess is cleaned up.

Seems the mass of synced containers broke something on the back end.

stoically / temporary-containers

Temporary containers not removed with new sync feature #371