swathipil
commented
2 years ago Hi @dakooka - thanks for reporting the issue! Would you be able to add the snippet of code that causes this issue? Thanks!
Closed dakooka closed 2 years ago
swathipil
commented
2 years ago Hi @dakooka - thanks for reporting the issue! Would you be able to add the snippet of code that causes this issue? Thanks!
dakooka
commented
2 years ago Hi @swathipil ,
Thank You for Your quick reaction
Unfortunately, there is nothing to add, since it does not work from the modelimport line.
from azure.mgmt.storage import models
On the bottom of the traceback (which I added also) we have found that the problem is occurring with
entire keyvaults module when it's trying to import Deserializer and Serializer
from msrest import Deserializer, Serializer
in _key_vault_client.py and _operations_mixin.py
Deserializer and Serializer are not used in these files at all
Next module which is failing is azure.mgmt for keyvault
from azure.common import credentials
from azure.mgmt import resource
from azure.mgmt import subscription
from azure.mgmt import storage
from azure.mgmt import keyvault
return keyvault.KeyVaultManagementClient(credentials=self.creds, subscription_id=self.subscription)
Error message
name 'Serializer' is not defined
It should failed for _key_vault_management_client.py from azure.mgmt module -> Serializer and Derializer are used there
PS Small remark we see one pull request in Your repo [Key Vault] Drop 2.7, 3.6 support (#22367) But a few days later the support was added back with the following pull request [Key Vault] Add back 3.6 support (#22556) Maybe it could broke module for using with python 3.6 version, I don't know
mccoyp
commented
2 years ago Hi @dakooka, thank you for opening an issue! This seems like a packaging error. msrest 0.6.21 does expose Serializer and Deserializer, and I'm able to successfully run
from msrest import Serializer, Deserializerand
import azure.keyvault.secretswith msrest==0.6.21 and azure-keyvault-secrets==4.3.0 installed, on Python 3.6.
Could I ask you to run pip freeze in the environment that had this issue, and share the output?
dakooka
commented
2 years ago Hello @mccoyp,
Here I'm pasting python modules which are present on our gitllab runner VM:
[gitlab-runner@amca-azure-devops terraform-profiles]$ python -m pip freeze adal==1.2.6 antlr4-python3-runtime==4.7.2 appdirs==1.4.4 applicationinsights==0.11.9 argcomplete==1.10.0 asn1crypto==0.24.0 azure-appconfiguration==1.1.1 azure-batch==10.0.0 azure-cli==2.19.1 azure-cli-command-modules-nspkg==2.0.3 azure-cli-core==2.19.1 azure-cli-nspkg==3.0.4 azure-cli-telemetry==1.0.6 azure-common==1.1.23 azure-core==1.9.0 azure-cosmos==3.1.1 azure-datalake-store==0.0.51 azure-functions-devops-build==0.0.22 azure-graphrbac==0.60.0 azure-identity==1.5.0 azure-keyvault==1.1.0 azure-keyvault-administration==4.0.0b1 azure-keyvault-secrets==4.2.0 azure-loganalytics==0.1.0 azure-mgmt-advisor==2.0.1 azure-mgmt-apimanagement==0.2.0 azure-mgmt-appconfiguration==1.0.1 azure-mgmt-applicationinsights==0.1.1 azure-mgmt-authorization==0.61.0 azure-mgmt-batch==9.0.0 azure-mgmt-batchai==2.0.0 azure-mgmt-billing==1.0.0 azure-mgmt-botservice==0.3.0 azure-mgmt-cdn==5.2.0 azure-mgmt-cognitiveservices==6.3.0 azure-mgmt-compute==18.2.0 azure-mgmt-consumption==2.0.0 azure-mgmt-containerinstance==1.5.0 azure-mgmt-containerregistry==3.0.0rc16 azure-mgmt-containerservice==9.4.0 azure-mgmt-core==1.2.1 azure-mgmt-cosmosdb==1.0.0 azure-mgmt-databoxedge==0.2.0 azure-mgmt-datalake-analytics==0.2.1 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-datamigration==4.1.0 azure-mgmt-deploymentmanager==0.2.0 azure-mgmt-devtestlabs==4.0.0 azure-mgmt-dns==2.1.0 azure-mgmt-eventgrid==3.0.0rc7 azure-mgmt-eventhub==4.1.0 azure-mgmt-hdinsight==2.2.0 azure-mgmt-imagebuilder==0.4.0 azure-mgmt-iotcentral==3.0.0 azure-mgmt-iothub==0.12.0 azure-mgmt-iothubprovisioningservices==0.2.0 azure-mgmt-keyvault==8.0.0 azure-mgmt-kusto==0.3.0 azure-mgmt-loganalytics==8.0.0 azure-mgmt-managedservices==1.0.0 azure-mgmt-managementgroups==0.2.0 azure-mgmt-maps==0.1.0 azure-mgmt-marketplaceordering==0.2.1 azure-mgmt-media==2.2.0 azure-mgmt-monitor==2.0.0 azure-mgmt-msi==0.2.0 azure-mgmt-netapp==0.15.0 azure-mgmt-network==17.0.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-policyinsights==0.5.0 azure-mgmt-privatedns==0.1.0 azure-mgmt-rdbms==3.1.0rc1 azure-mgmt-recoveryservices==0.4.0 azure-mgmt-recoveryservicesbackup==0.11.0 azure-mgmt-redhatopenshift==0.1.0 azure-mgmt-redis==7.0.0rc2 azure-mgmt-relay==0.1.0 azure-mgmt-reservations==0.6.0 azure-mgmt-resource==12.0.0 azure-mgmt-search==8.0.0 azure-mgmt-security==0.6.0 azure-mgmt-servicebus==0.6.0 azure-mgmt-servicefabric==0.5.0 azure-mgmt-signalr==0.4.0 azure-mgmt-sql==0.26.0 azure-mgmt-sqlvirtualmachine==0.5.0 azure-mgmt-storage==16.0.0 azure-mgmt-subscription==0.2.0 azure-mgmt-synapse==0.6.0 azure-mgmt-trafficmanager==0.51.0 azure-mgmt-web==0.48.0 azure-multiapi-storage==0.5.2 azure-nspkg==3.0.2 azure-storage-blob==1.5.0 azure-storage-common==1.4.2 azure-synapse-accesscontrol==0.2.0 azure-synapse-artifacts==0.3.0 azure-synapse-spark==0.2.0 bcrypt==3.1.7 blessings==1.7 bpython==0.20.1 certifi==2019.9.11 cffi==1.12.3 chardet==3.0.4 colorama==0.4.1 cryptography==2.7 curtsies==0.3.4 distlib==0.3.1 fabric==2.5.0 filelock==3.0.12 greenlet==0.4.17 humanfriendly==4.18 idna==2.8 importlib-metadata==3.4.0 importlib-resources==5.1.0 invoke==1.3.0 isodate==0.6.0 javaproperties==0.5.1 Jinja2==2.10.1 jmespath==0.9.4 jsmin==2.2.2 jsondiff==1.2.0 knack==0.8.0rc2 MarkupSafe==1.1.1 mock==2.0.0 msal==1.6.0 msal-extensions==0.3.0 msrest==0.6.10 msrestazure==0.6.4 oauthlib==3.1.0 paramiko==2.6.0 pbr==5.4.3 pkginfo==1.7.0 portalocker==1.5.1 psutil==5.8.0 pycparser==2.19 pydocumentdb==2.3.5 Pygments==2.4.2 pyhcl==0.4.4 PyJWT==1.7.1 PyNaCl==1.3.0 pyodbc==4.0.30 pyOpenSSL==19.0.0 python-dateutil==2.8.0 python-terraform==0.10.1 pytz==2019.1 PyYAML==5.1.2 requests==2.22.0 requests-oauthlib==1.2.0 scp==0.13.2 six==1.12.0 sshtunnel==0.1.5 tabulate==0.8.5 typing-extensions==3.7.4.3 urllib3==1.25.11 virtualenv==20.4.2 vsts==0.1.25 vsts-cd-manager==1.0.2 wcwidth==0.2.5 websocket-client==0.56.0 xmltodict==0.12.0 zipp==3.4.0
Thank You and have a great day, Rosty
mccoyp
commented
2 years ago Hi @dakooka, thank you for the quick response. I've tried using the same versions of every package mentioned in the stack trace, but I haven't been able to reproduce the error. Below is the output of running pip freeze in my environment:
adal==1.2.6
azure-common==1.1.23
azure-core==1.9.0
azure-identity==1.5.0
azure-keyvault==1.1.0
azure-keyvault-secrets==4.2.0
azure-mgmt-core==1.2.1
azure-mgmt-keyvault==8.0.0
azure-mgmt-storage==16.0.0
azure-nspkg==3.0.2
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.11
cryptography==3.4.8
idna==3.3
isodate==0.6.1
msal==1.6.0
msal-extensions==0.3.0
msrest==0.6.10
msrestazure==0.6.4
oauthlib==3.1.0
portalocker==1.7.1
pycparser==2.21
PyJWT==1.7.1
python-dateutil==2.8.2
python-terraform==0.10.1
pywin32==303
requests==2.27.1
requests-oauthlib==1.2.0
six==1.16.0
urllib3==1.26.8Running from msrest import Deserializer, Serializer still doesn't raise an error. Neither do from azure.mgmt.storage import models, import azure.keyvault.secrets, from azure.keyvault.secrets import _models, or
from azure.identity import DefaultAzureCredential
from azure.mgmt import keyvault
client = keyvault.KeyVaultManagementClient(credential=DefaultAzureCredential(), subscription_id="")It seems to me that the msrest package installed in your environment must be the issue. Are you able to manually run from msrest import Deserializer, Serializer successfully? If you pip uninstall msrest and try reinstalling, does that help at all?
dakooka
commented
2 years ago Hello @mccoyp
We tried to reinstall msrest package but we guess we found where the possible problem could be. When we are running directly from VM level everything is working:
But when we are trying to run with creating python environment (the screenshots above are from the gitlab pipeline output) it does not work:
Following command which we run:
virtualenv -p python3 .venv-test
source .venv-test/bin/activate
pip install -r tools/requirements.txt
python tools/manage.py -h -> and it's failing with the provided error message
mccoyp
commented
2 years ago @dakooka Hm... well, it's good to see that the command works at the VM level! As for the Python environment, looking at the last screenshot you provided, I think it's possible that the virtual environment you created isn't actually active; there's no (.venv-test) indicator on the command line that I can see. If you run
source .venv-test/bin/activate
pip install -r tools/requirements.txtand then run pip freeze, do you see the packages you expect to see? Or, is it possible that you need to run pip install -r requirements.txt from inside the tools directory? It looks like the successful command from the VM was run from that directory.
ghost
commented
2 years ago Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!
Describe the bug We are using some python modules in our python scripts, and when gitlab-runner is trying to run test for this python script we are getting error message: from msrest import Deserializer, Serializer ImportError: cannot import name 'Deserializer'
requirements.txt We are also tried to use the last versions of the Python Azure SDK and clear the caches for gitlab-runner but the issue is still present
Exception or Stack Trace
Setup (please complete the following information):
Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added