[bug]: External connections to Cloudfront when switching models?

[ghost]

Is there an existing issue for this?

• [X] I have searched the existing issues

OS

Linux

GPU

cuda

VRAM

No response

What version did you experience this issue on?

2.3.5

What happened?

Recently my installation of InvokeAI has begun opening external connections when i switch models (does not matter which model). The connections go to Cloudfront and since it's HTTPS i can't really see what's up with Wireshark. The connection stays up for the whole time InvokeAI is loaded and running. What is the reason behind the connections and how do i toggle it off, ensuring none are made?

I'm fairly sure this didn't happen in previous versions.

Screenshots

No response

Additional context

No response

Contact Details

No response

[ghost]

Adding a screenshot. I use "watch -n 1 lsof -i" to see real-time active connections.

[damian0815]

this seems bad and certainly not intended behaviour afaik. what version did you upgrade from?

[ghost]

this seems bad and certainly not intended behaviour afaik. what version did you upgrade from?

Upgraded yesterday from 2.3.5 rc3, if i remember correctly. Before that i had 2.3.4 post1.

Can someone reproduce the connection? For what it's worth, i only download safetensors models from CivitAI or Huggingface, i haven't used the unsafe format in maybe 3-4 months. I convert the models i like to Diffusers, slap on a VAE from a local dir, and generate away. A malicious model seems unlikely, unless something was just renamed to safetensors and escaped CivitAI/Huggingface scanning.

edit: further testing; upon launching InvokeAI no connections are made. Only the localhost listening address. The Cloudfront connection is opened when i open the web interface.

It seems i don't need to (re)load a model after all for it to appear, but the connection is clearly refreshed when i do.

[hipsterusername]

I doubt this is a malicious model, but would not be surprised if there's a connection to Huggingface that is happening somewhere in the pipeline.

Are you pulling in anything from Huggingface using the repo id directly?

[ghost]

I doubt this is a malicious model, but would not be surprised if there's a connection to Huggingface that is happening somewhere in the pipeline.

Are you pulling in anything from Huggingface using the repo id directly?

The models i use are all saved locally, with paths to their respective folders on my drives for both the converted model files and the VAE folder.

The only models that have references to Huggingface ID's in the model list are the original Stable Diffusion 1.5 and 2.1, e.g. "runwayml/stable-diffusion-v1-5"

InvokeAI loads the last used model on launch, meaning neither of those is never loaded when i start InvokeAI since i don't use them unless i want to mess with TI training etc.

[psychedelicious]

I'll just note that the web UI is 100% local. It communicates with the web server only. Zero analytics or external network communications.

Same for the web server. I suspect it's the HF hub?

[ghost]

I'll just note that the web UI is 100% local. It communicates with the web server only. Zero analytics or external network communications.

Same for the web server. I suspect it's the HF hub?

Thanks, i don't suspect anything intentional here. Just got curious because i figured that's how it's supposed to be; no external connections.

[hipsterusername]

@lstein - Is the HF hub getting pinged for anything even when users aren't actively using HF models?

I'm wondering if this related to autocompletes for textual inversion/concept library...

[ghost]

For what it's worth, "Show Textual Inversions from HF Concepts Library" is disabled in my settings.

[psychedelicious]

I can confirm that there is a connection to cloudfront on both nodes web and CLI. The connection is made during model loading. Also, HF's CDN is cloudfront so I guess this is something related to that.

If it's just to get the TI embeddings, I would have expected a transient connection, but this seems to persist.

[ghost]

Seems so, the connection does go to a CLOSED_WAIT state after a while but it re-establishes as soon as i switch models. Also, naturally, if the HF Concept Library toggle is turned off, one would expect no connection to be made at all.

[psychedelicious]

I did a bit of sleuthing. The connection is indeed made by HF diffusers, specifically when calling AutoencoderKL.from_pretrained(). It's possible that other methods would also make outbound connections, but this is the first thing we do in the code to load a model.

Edit: Yeah, I guess this is expected. I don't like it, though, and the "offline mode" link is dead.

[divens]

Hey there, did some digging and found this here

Our library gathers telemetry information during from_pretrained() requests. This data includes the version of Diffusers and PyTorch/Flax, the requested model or pipeline class, and the path to a pretrained checkpoint if it is hosted on the Hub.

We understand that not everyone wants to share additional information, and we respect your privacy, so you can disable telemetry collection by setting the DISABLE_TELEMETRY environment variable

Searched the repo and didn't find any reference to it, so maybe it's worth a try

[lstein]

@lstein - Is the HF hub getting pinged for anything even when users aren't actively using HF models?

I'm wondering if this related to autocompletes for textual inversion/concept library...

This is very likely the cause of the problem. If the web interface is set to add HuggingFace textual inversion concepts library terms to the "Textual Inversion" popup menu, then Invoke will indeed ping HuggingFace to download the terms. It should only be doing this once per session, however, and if it goes to HF each time a model is changed then this is a bug.

[lstein]

Hey there, did some digging and found this here

Our library gathers telemetry information during from_pretrained() requests. This data includes the version of Diffusers and PyTorch/Flax, the requested model or pipeline class, and the path to a pretrained checkpoint if it is hosted on the Hub.

We understand that not everyone wants to share additional information, and we respect your privacy, so you can disable telemetry collection by setting the DISABLE_TELEMETRY environment variable

Searched the repo and didn't find any reference to it, so maybe it's worth a try

We try to minimize the need for an active internet conenction while InvokeAI is running, which is why there is a model install step during the initial setup and configuration. All our standard models are from HuggingFace, however, so during the install there will be connections to HuggingFace (presumably via cloudflare). Of course, if you use the model manager, model installer, or CLI to download more models from HuggingFace or Civitai (or elsewhere), there will again be connections.

I do find a DISABLE_TELEMETRY environment variable in the diffusers hub_utils.py file. What it seems to do is to add an additional argument to the request passed to the REST endpoint when a model is requested for download. It doesn't look like setting it will prevent the connections, but they will prevent HF from logging your requests (or whatever it is they're doing with the data).

[lstein]

@blessedcoolant [TLDR] Whenever the model is changed and the HF Concepts Library is enabled, InvokeAI reaches out to HuggingFace to download the list of concept trigger terms. It should only need to do this once at the beginning of each session. I've tracked the problem down to this bit of code in the web backend:

        @socketio.on("getTextualInversionTriggers")
        def get_ti_triggers():
            try:
                local_triggers = self.generate.model.textual_inversion_manager.get_all_trigger_strings()
                locals = [{'name': x} for x in sorted(local_triggers, key=str.casefold)]
                concepts = HuggingFaceConceptsLibrary().list_concepts(minimum_likes=5)
                concepts = [{'name': f'<{x}>'} for x in sorted(concepts, key=str.casefold) if f'<{x}>' not in local_triggers]
                socketio.emit("foundTextualInversionTriggers", {'local_triggers': locals, 'huggingface_concepts': concepts})
            except Exception as e:
                self.handle_exceptions(e)

I can fix this by caching the HuggingFaceConceptsLibrary object as a module variable in invoke_ai_web_server or by keeping a singleton object in concepts_lib. Which is the better pattern?

[EDIT] I changed concepts_lib to return a singleton object, which caches the hugging face concepts and does not perform redundant fetches. I haven't fixed the problem of the terms being download even when the user has requested they not be displayed.

[lstein]

@blessedcoolant Another issue: The web interface is downloading the concept terms even when "show textual inversion terms from HF" is disabled. It looks like get_ti_triggers() always asks for the list of concepts.

[psychedelicious]

@lstein I stepped through the code in a debugger and the connection is made when calling from_pretrained. Noted here https://github.com/invoke-ai/InvokeAI/issues/3330#issuecomment-1533100569

It's also documented in the diffusers docs.

It's not the concepts connection, that just makes single requests.

[lstein]

Nice detective work. I'll put together a small test script that illustrates the problem and file a bug report against the appropriate HuggingFace repository.

Can you tell whether the issue is in the diffusers from_pretrained() method specifically, or is it a deeper problem with HuggingFace's hub API? Naively I'd expect the latter.

[psychedelicious]

I didn't dig into the diffusers code except briefly but I'd also guess it's in the hub. The file in which from_pretrained is defined doesn't appear to import any network modules except the hub.

[divens]

Thanks to you guys for looking into this.

We try to minimize the need for an active internet conenction while InvokeAI is running, which is why there is a model install step during the initial setup and configuration. All our standard models are from HuggingFace, however, so during the install there will be connections to HuggingFace (presumably via cloudflare). Of course, if you use the model manager, model installer, or CLI to download more models from HuggingFace or Civitai (or elsewhere), there will again be connections.

Yes for sure, the expected behavior would be that when all the init step is done (including first download of models), there are no outbound connections anymore (as you said, except if the user himself interacts with third party model sources).

Searched the repo and didn't find any reference to it, so maybe it's worth a try

I do find a DISABLE_TELEMETRY environment variable in the diffusers hub_utils.py file. What it seems to do is to add an additional argument to the request passed to the REST endpoint when a model is requested for download. It doesn't look like setting it will prevent the connections, but they will prevent HF from logging your requests (or whatever it is they're doing with the data).

Yes I meant I didn't find this environment variable in Invoke repo. Since this is an opt out on the side of HF, I think it would be nice to include a heads up in Invoke documentation, so people are aware of its existence and can act accordingly would they prefer not to participate in HF telemetry.

[cmawhorter]

this was causing me issues on a firewalled server w/o direct network access. invoke would seem to hang often.

i tried disable_telemetry and hf offline mode flags and it didn't fix it. but doing echo '127.0.0.1 huggingface.co' >> /etc/hosts finally makes things work.

i can't find it atm, but i've seen a local_files_only=True or something mentioned in diffusers issues. maybe that should be the default in invoke?

Edit: not sure why, but for some reason that alone isn't working on my local because of the HF concepts lib call. if you're on 2.3.5, edit backend/invoke_ai_web_server.py (e.g. "vi ./.venv/lib/python3.9/site-packages/invokeai/backend/invoke_ai_web_server.py") and replace line 542 with this concepts = []

and fwiw -- i'm launching invokeai with DISABLE_TELEMETRY=1 HF_HUB_OFFLINE=1 invokeai --no-internet in case that's also helping things work. :shrug:

...and found it, local_files_only=True