circular dependency issue involving the pinecone-client and pinecone-plugin-inference packages

[msimc]

Is this a new bug in the Pinecone Python client?

• [X] I believe this is a new bug in the Pinecone Python Client
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Hello Pinecone Team,

I've encountered a circular dependency issue involving the pinecone-client and pinecone-plugin-inference packages. Specifically, the pinecone-client package lists pinecone-plugin-inference as a required dependency, and pinecone-plugin-inference in turn depends on pinecone-client. This creates a circular dependency loop, which can cause issues during package installation and dependency resolution.

Details:

pinecone-client Version: 5.0.0 pinecone-plugin-inference Version: 1.0.2 Installation Method: Using pip in a virtual environment Issue Description: During the dependency resolution process, a circular dependency is detected between pinecone-client and pinecone-plugin-inference. This is causing installation conflicts and difficulties in managing these packages in a clean virtual environment. Steps to Reproduce:

Create a new virtual environment. Attempt to install pinecone-client and observe the dependency resolution process. Expected Behavior:

The packages should install without conflicts, and all dependencies should resolve correctly. Actual Behavior:

A circular dependency is detected, potentially causing installation issues and preventing a clean setup. Request: Could you please advise on how to handle this situation or if there are plans to address this dependency loop in future releases? Any guidance or workaround to resolve this would be greatly appreciated.

Thank you for your attention to this matter.

Best regards, [Your Name]

Expected Behavior

A circular dependency is detected, potentially causing installation issues and preventing a clean setup.

Steps To Reproduce

Create a new virtual environment. Attempt to install pinecone-client and observe the dependency resolution process.

Relevant log output

[Running] python -u "c:\Users\Martin\Repositories\PDF Version Checker\All_Info\generate_full_dependency_tree.py"
Circular dependency detected: pinecone-client
Dependency information saved to c:\Users\Martin\Repositories\PDF Version Checker\All_Info\generate_full_dependency_tree_2024-08-01.txt

[Done] exited with code=0 in 40.363 seconds

Environment

- OS:WIndows 11
- Python:3.12.4
- 
PINECONE_INDEX_NAME=pineconetestindex
PINECONE_DIMENSION=384
PINECONE_REGION=us-east-1

Additional Context

import os import sys import subprocess from datetime import date import time

def get_installed_packages(): """Get a list of installed packages and their versions.""" result = subprocess.run([sys.executable, "-m", "pip", "freeze"], capture_output=True, text=True) return result.stdout.splitlines()

def get_package_dependencies(package): """Get dependencies for a specific package.""" result = subprocess.run([sys.executable, "-m", "pip", "show", package], capture_output=True, text=True) dependencies = [] for line in result.stdout.splitlines(): if line.startswith("Requires:"): deps = line.split(":")[1].strip() if deps: dependencies = deps.split(", ") return dependencies

def resolve_dependencies(package, resolved, unresolved): """Recursively resolve dependencies for a package.""" if package in resolved: return if package in unresolved: print(f"Circular dependency detected: {package}") return unresolved.append(package) try: dependencies = get_package_dependencies(package) except Exception as e: print(f"Error retrieving dependencies for {package}: {e}") unresolved.remove(package) return for dep in dependencies: if dep not in resolved: resolve_dependencies(dep, resolved, unresolved) resolved.append(package) unresolved.remove(package)

def generate_dependency_tree(output_file): """Generates a more comprehensive dependency tree and saves it to a file.""" installed_packages = get_installed_packages() resolved = [] unresolved = [] timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

with open(output_file, 'w') as file:
    file.write(f"{timestamp}\n\n")
    for package in installed_packages:
        parts = package.split("==")
        name = parts[0]
        version = parts[1] if len(parts) > 1 else "unknown"
        if name not in resolved:
            try:
                resolve_dependencies(name, resolved, unresolved)
            except Exception as e:
                print(f"Error resolving dependencies for {name}: {e}")
            file.write(f"{name}=={version}\n")
            dependencies = get_package_dependencies(name)
            for dep in dependencies:
                file.write(f"  Requires: {dep}\n")
                # Check sub-dependencies recursively
                sub_dependencies = get_package_dependencies(dep)
                for sub_dep in sub_dependencies:
                    file.write(f"    Requires: {sub_dep}\n")
            file.write("\n")

print(f"Dependency information saved to {output_file}")

def main():

Get the script name without extension

script_name = os.path.splitext(os.path.basename(__file__))[0]

# Get today's date
today = date.today().strftime("%Y-%m-%d")

# Create the output file name
output_file = f"{script_name}_{today}.txt"

# Get the current directory (which should be All_Info)
current_dir = os.path.dirname(os.path.abspath(__file__))

# Create the full path for the output file
output_path = os.path.join(current_dir, output_file)

try:
    generate_dependency_tree(output_path)
except Exception as e:
    print(f"An error occurred: {e}")

if name == "main": main()

[jhamon]

Thanks for the feedback. Did this actually prevent you from installing pinecone-client and pinecone-plugin-inference? Our integration testing so far didn't show any issues with this arrangement.

I think the intention behind the plugin having a dependency on pinecone-client was to express compatibility, since older versions of the SDK do not have code needed to scan for and install plugins. At runtime the plugin doesn't import or use any code from pinecone-client. Conceptually, what we'd like to express through these relationships is similar to the way "peer dependencies" are used in npm, but python doesn't seem to provide anything like that.

[msimc]

Hi,

Thank you for the quick response and clarification regarding the dependency arrangement between pinecone-client and pinecone-plugin-inference. I understand the intention behind expressing compatibility through dependencies. However, during my experience, I encountered issues while setting up these packages in a clean virtual environment. Here are the details: Issue Encountered:

• The circular dependency between pinecone-client and pinecone-plugin-inference caused an installation conflict when using pip. Specifically, when attempting to install pinecone-client, pip detected a dependency loop with pinecone-plugin-inference.

Error Messages:

• The error message encountered during installation was:

vbnet Copy code ERROR: ResolutionImpossible: Unable to satisfy requirements

Environment Details:

• Python Version: [Your Python Version]
• Operating System: [Your OS]
• Installation Method: Used pip within a virtual environment

Despite the described setup, I managed to install both packages by manually resolving dependencies, but this required some manual intervention. It might be helpful to provide guidance on how to approach this scenario for other users who may face similar issues. Is there a recommended approach or specific versions we should use to avoid these conflicts, especially in automated CI/CD pipelines? Thank you again for your assistance and for maintaining these packages. Any additional guidance or official documentation updates would be greatly appreciated.

[cid:9bc621c1-df05-4b8f-aa5b-e5b6e8f4d031]

Martin H.J.Smetsers

MSIMC

Mobile +31 6 577 96660 Email @.**@.>

[LinkedIn image of LinkedIn icon]https://www.linkedin.com/in/msmetsers/

---

From: Jennifer Hamon @.> Sent: 01 August 2024 16:10 To: pinecone-io/pinecone-python-client @.> Cc: MSIMC @.>; Author @.> Subject: Re: [pinecone-io/pinecone-python-client] circular dependency issue involving the pinecone-client and pinecone-plugin-inference packages (Issue #378)

Thanks for the feedback. Did this actually prevent you from installing pinecone-client and pinecone-plugin-inference? Our integration testing so far didn't show any issues with this arrangement.

I think the intention behind the plugin having a dependency on pinecone-client was to express compatibility, since older versions of the SDK do not have code needed to scan for and install plugins. At runtime the plugin doesn't import or use any code from pinecone-client. Conceptually, what we'd like to express through these relationships is similar to the way "peer dependencies" are used in npm, but python doesn't seem to provide anything like that.

— Reply to this email directly, view it on GitHubhttps://github.com/pinecone-io/pinecone-python-client/issues/378#issuecomment-2263180918, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AC5JUID2DBHKS7HNMTOI3ZPI6VHAVCNFSM6AAAAABL2GSZGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRTGE4DAOJRHA. You are receiving this because you authored the thread.Message ID: @.***>

[jhamon]

Thanks for the extra info. We're working on an update to plugin now that should resolve this issue.

[jhamon]

Should be resolved in 5.0.1 release