Issue with updating 'sites' with odd characters in values

[bdlamprecht]

Issue type

[X] Bug report

Environment

• Python version: 3.5
• NetBox version: 2.3.3
• PyNetBox version: 3.3.1

Description

I'm able to create new sites using pynetbox with the method nb.dcim.sites.create(**data) and data including all of the required fields such as name, slug, status, and region(that last one isn't technically "required", but I'm using it nonetheless).

However, when I try to update existing sites with the following code:

nb_site = nb.dcim.sites.get(name="00j")
nb_site.physical_address = "C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES"
nb_site.save()

I get the following HTML error:

int() argument must be a string, a bytes-like object or a number, not 'dict'

I can do the same action with a curl request just fine (sent via a PUT to /api/dcim/sites/434/):

{
  "name": "00j",
  "slug": "00j",
  "status": 1,
  "physical_address": "C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES"
}

Results in this response:

{
  "id": 434,
  "name": "00j",
  "slug": "00j",
  "status": 1,
  "region": 375,
  "tenant": null,
  "facility": "",
  "asn": null,
  "time_zone": "Europe/Madrid",
  "description": "WHOIS details were updated on 'June 06 2018 22:32:54 UTC'",
  "physical_address": "C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES",
  "shipping_address": "",
  "contact_name": "",
  "contact_phone": "",
  "contact_email": "",
  "comments": "",
  "custom_fields": {},
  "created": "2018-06-07",
  "last_updated": "2018-06-07T06:57:56.860491Z"
}

At first I thought it was due to an issue with the "odd" characters in the physical_address, but it appears to be more widespread than that (even values without the "odd" characters appears to also fail). Perhaps I'm doing something obviously wrong (wouldn't be the first time), but I'm not quite sure if that is the case in this scenario.

If you need further information from me, please let me know.

[bdlamprecht]

I thought I would add a few more details that I discovered while trying to understand the depth of this issue.

In the WebUI, I can see the following text for the physical_address:

However, using pynetbox to get the same information leads to a different (I believe) result:

>>> nb_site = nb.dcim.sites.get(434)
>>> nb_site
00j
>>> nb_site.physical_address
'C/ SAKURA, 8\nSAN FRUITĂ\x92S DEL BALGĂ\x88S, B, 08202, ES'
>>>

If you'll notice the special "O" in the word "FRUITOS" becomes Ă\x92 and the special "E" in the word "BALGES" becomes Ă\x88.

I'm not an expert on Unicode so that could be correct, I just thought I would point it out as a possible issue.

[bdlamprecht]

Closing this issue after opening #63.

[bdlamprecht]

Well, after further investigation, it also appears that the odd characters ARE causing issues. As an example, the following code doesn't work for comparison:

if nb_site.physical_address != full_address:
    update = True

    # DEBUG
    print("Address doesn't match")

# FURTHER LOGIC TO COMPARE OTHER FIELDS NOT INCLUDED

if update:
    nb_site.physical_address = address
    nb_site.time_zone = timezone
    nb_site.custom_fields['bgp_comm'] = bgp_comm
    nb_site.custom_fields['reso_code'] = reso_code

try:
    nb_site.save()
    nb_updated_sites.append(nb_site.name)
except pynetbox.lib.query.RequestError as e:
    print("Failed due to:\n    {}".format(e.error))

The value of full_address is as follows:

C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES

This doesn't cause an issue when full_address contains just "normal" characters such as:

6300 DIAGONAL HWY\nBOULDER, CO, 80301-6108, US

[zachmoody]

I was able to have a look at this, it does seem that .save() isn't idempotent when non-ascii characters are involved. Which isn't good, but shouldn't be fatal.

I usually see this error when NetBox is expecting an int referencing an index in a related field, but gets the dict response from the nested serializer. Pynetbox should be pulling the id field from that and sending that when .serialize() is called as a part of .save().

I've got a test that reproduces the behavior with the non-ascii characters, and I'll work on a fix there. Not sure it'll help you here and in #63 though. I'm curious what .serialize() returns if you call it in your scripts.

[bdlamprecht]

Good to know that you were able to reproduce at least a part of the issue that was reported. Very glad to hear that I wasn't doing something really incorrect (although that isn't off the table just yet :smile: )

It would be great if you could "work on a fix there".

As requested, here is the output from .serialize() on the object that has the non-ascii characters in the address:

{'status': 1, 'id': 396, 'tenant': None, 'created': '2018-06-20', 'count_devices': 0, 'time_zone': 'Europe/Madrid', 'contact_email': '', 'custom_fields': {'ce_qos_type': {'label': '', 'value': None}, 'site_2nd_asn': None, 'coordinates': '041.77758468, 001.86634540', 'gps_id': None, 'reso_code': '00J', 'bgp_comm': '163:49210'}, 'asn': None, 'contact_phone': '', 'count_racks': 0, 'shipping_address': '', 'last_updated': '2018-06-20T21:49:28.432012Z', 'count_prefixes': 0, 'count_circuits': 0, 'comments': '', 'name': '00j', 'contact_name': '', 'physical_address': 'C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES', 'description': "Site information was updated with WHOIS details on 'June 21 2018 05:56:03 UTC'", 'region': 365, 'count_vlans': 0, 'facility': '', 'slug': '00j'}

[zachmoody]

Hrm, looks like it's properly preparing it for NetBox's API. Any chance you still get this error when custom_fields aren't involved?

[bdlamprecht]

Well, I'm doing comparisons, between what is already in NetBox and what is being updated in NetBox, the only thing that differs is the address field.

Such as in this example:

# Check if address is correct
if nb_site.physical_address != full_address:
    update = True
    fields_to_update.append("Address")

The above if statement returns True when there are only ascii-characters in nb_site.physical_address: However, once you start adding in non-ascii characters, the if statement returns False.

The code I'm using:

# DEBUG
print("site in netbox '{}'\n".format(nb_site.name))
print("physical_address already in netbox is:\n{}\n".format(nb_site.physical_address))
print("full_address attempting to be updated is:\n{}".format(full_address))

Site with only ascii characters in the address shows the following output:

site in netbox 'bld'

physical_address already in netbox is:
6300 DIAGONAL HWY
BOULDER, CO, 80301-6108, US

full_address attempting to be updated is:
6300 DIAGONAL HWY
BOULDER, CO, 80301-6108, US

Site with non-ascii chacters in the address shows the following output:

site in netbox '00j'

physical_address already in netbox is:
C/ SAKURA, 8
SAN FRUIT�S DEL BALG�S, B, 08202, ES

full_address attempting to be updated is:
C/ SAKURA, 8
SAN FRUITÒS DEL BALGÈS, B, 08202, ES

So, either NetBox or pynetbox is doing something weird with the data, because the information hasn't changed from when it was first imported, I'm just merely checking to see if when information changes, the code still works.

[zachmoody]

:disappointed: So, I goofed up when I said I had a test that reproduced the behavior. Turns out I'd copy/pasta'd an assertFalse where it should've been an assertTrue. :|

So can't reproduce this, but #70 had a similar 500 from NetBox. Looks like you're using custom fields in at least some of of your examples. Just don't know if they're selection types or not. Can you give it a shot with the new pynetbox version for fun?

[bdlamprecht]

Nope, looks like it isn't fixed:

$ pip freeze | grep pynetbox
pynetbox==3.4.2

Running the same code in the terminal:

site in netbox '00j'

physical_address already in netbox is:
C/ SAKURA, 8
SAN FRUIT�S DEL BALG�S, B, 08202, ES

full_address attempting to be updated is:
C/ SAKURA, 8
SAN FRUITÒS DEL BALGÈS, B, 08202, ES

To be clear, the physical_address field displays correctly in both the WebUI and the API. The only place where it doesn't work is in the terminal and the code for comparision (i.e. if nb_site.physical_address != full_address).

Snapshot of the WebUI:

Information pulled from the API at http://netbox.example.com/api/dcim/sites/387/:

{
  "id": 387,
  "name": "00j",
  "slug": "00j",
  "status": {
    "value": 1,
    "label": "Active"
  },
  "region": {
    "id": 365,
    "url": "http://netbox.example.com/api/dcim/regions/365/",
    "name": "San Fruitos",
    "slug": "san-fruitos-es"
  },
  "tenant": null,
  "facility": "",
  "asn": null,
  "time_zone": "Europe/Madrid",
  "description": "Site information was updated with WHOIS details on 'July 10 2018 22:12:48 UTC'",
  "physical_address": "C/ SAKURA, 8\nSAN FRUITÒS DEL BALGÈS, B, 08202, ES",
  "shipping_address": "",
  "contact_name": "",
  "contact_phone": "",
  "contact_email": "",
  "comments": "",
  "custom_fields": {
    "reso_code": "00J",
    "bgp_comm": "163:49210",
    "ce_qos_type": {
      "value": null,
      "label": ""
    }
  },
  "created": "2018-07-09",
  "last_updated": "2018-07-10T14:15:14.653299Z",
  "count_prefixes": 2,
  "count_vlans": 0,
  "count_racks": 0,
  "count_devices": 0,
  "count_circuits": 0
}

I'm not sure what the difference is in our setups are since you can't seem to reproduce this issue, but I can do so fairly easily. Again, I'm guessing the issue is with the handling of UTF in pynetbox, but cannot guarantee it.

What else do you need from me to further troubleshoot this issue?

[zachmoody]

There's at least two issues here. The one that's, to me, more clearly in pynetbox's domain is the 500 from NetBox:

int() argument must be a string, a bytes-like object or a number, not 'dict'

That's the same error was happening in #70 when selection type custom fields, that aren't modified, are passed back via PUTs as a dict an causing NetBox to throw 500s. Which I see you have.

The second problem, of why the comparison operation in your script is failing. I'm sure is due to some unicode thing I don't fully grok yet (if someone else might know, feel free to chime in). But I think you can probably get around it with some combination of .encode() or .decode() on either the string from pynetbox or your source file. fwiw, I get different results depending on what OS and term app I'm using. OSX gives me SAN FRUIT�S DEL BALG�S, B, 08202, ES Linux I get SAN FRUITÒS DEL BALGÈS, B, 08202, ES from .physical_address.

[bdlamprecht]

So I tried all combinations of .encode() on the source of the information being passed to pynetbox as well as .decode() on the values stored in NetBox (previously populated using pynetbox), and I'm not able to get it to work:

Using .encode() on the source of the information produces the following:

full_address attempting to be updated is:
b'C/ SAKURA, 8\nSAN FRUIT\xc3\x92S DEL BALG\xc3\x88S, B, 08202, ES'

If I try and pass that to pynetbox, I get this error:

File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'C/ SAKURA, 8\nSAN FRUIT\xc3\x92S DEL BALG\xc3\x88S, B, 08202, ES' is not JSON serializable

If I try and do a combination of .encode().decode(), I get the same result as I originally reported.

Example after removing the physical_address completely in the WebUI

site in netbox '00j'

physical_address already in netbox is:

full_address attempting to be updated is:
C/ SAKURA, 8
SAN FRUITÒS DEL BALGÈS, B, 08202, ES

Attempting to update it again with the exact same code (.encode().decode()):

site in netbox '00j'

physical_address already in netbox is:
C/ SAKURA, 8
SAN FRUIT�S DEL BALG�S, B, 08202, ES

full_address attempting to be updated is:
C/ SAKURA, 8
SAN FRUITÒS DEL BALGÈS, B, 08202, ES

What I find interesting, is somehow, between when it is passed to NetBox and then later retrived from NetBox (both via pynetbox), the results are different (when viewed in their bytestring formats displayed with .encode().

site in netbox '00j'

physical_address already in netbox is:
b'C/ SAKURA, 8\nSAN FRUIT\xc4\x82\xc2\x92S DEL BALG\xc4\x82\xc2\x88S, B, 08202, ES'

full_address attempting to be updated is:
b'C/ SAKURA, 8\nSAN FRUIT\xc3\x92S DEL BALG\xc3\x88S, B, 08202, ES'

Using a simple text editor to compare the two unicode strings...

The Ò in FRUITÒS in NetBox is stored as:

\xc4\x82\xc2\x92

But in the source it is really:

\xc3\x92

(the value \xc3 appears to become \xc4\x82\xc2).

The same issue occurs with the È in BALGÈS. It is stored in NetBox as:

\xc4\x82\xc2\x88

But in the source it is really

\xc3\x88

(again, the value \xc3 appears to become \xc4\x82\xc2).

I have no idea what those escaped characters truly mean (I'll try and look into it, but I have zero experience in unicode to date). Hopefully, you (or someone else more experienced than I) can resolve the core of the issue.

[zachmoody]

:disappointed: Sorry for all the back and forth. I found the issue. I'll have a patch merged shortly.