search

mixpanel / mixpanel-utils

Other
87 stars 63 forks source link

Bug causing crash when importing from Amplitude #40

Closed dhpwd closed 2 years ago

dhpwd commented 2 years ago

Hi guys,

Getting a crash when using the import_from_amplitude method (see bottom of issue for full stack trace)

Key part is "line 1523 ... response_data = json.loads(response) ValueError: Expected object or value"

Worth noting that numerous requests succeed before the crash (and the data shows up in Mixpanel) so this would indicate we're dealing with an uncommon response value from the Mixpanel API 🤔

Let me know if I can provide more info to help!

Thanks,

Dan

Exception in thread Thread-350:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 592, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 776, in _set
    self._callback(self._value)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1523, in _async_response_handler_callback
    response_data = json.loads(response)
ValueError: Expected object or value
jaredmixpanel commented 2 years ago

@dhpwd this issue should now be fixed in v2.2.1.

dhpwd commented 2 years ago

@jaredmixpanel still seeing a crash I'm afraid (full log output below)

pip list | grep mixpanel returns mixpanel-utils 2.2.1 so library definitely up to date

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 5 items on 2022-01-27 10:54:44!
DEBUG: API Response: {"error":"a temporary failure occurred","status":0}
WARNING: API response NOT OK: {"error":"a temporary failure occurred","status":0}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":6,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":7,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":8,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":9,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":10,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":11,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":12,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":13,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":14,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":15,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":16,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":17,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":18,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":19,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":20,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":21,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":22,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":23,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":24,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":25,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":26,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":27,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 5 items on 2022-01-27 10:54:44!
DEBUG: API Response: {"error":"a temporary failure occurred","status":0}
WARNING: API response NOT OK: {"error":"a temporary failure occurred","status":0}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 5 items on 2022-01-27 10:54:44!
DEBUG: API Response: {"error":"a temporary failure occurred","status":0}
WARNING: API response NOT OK: {"error":"a temporary failure occurred","status":0}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":6,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":7,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":8,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":9,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":10,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":11,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":12,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":13,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":14,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":15,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":16,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":17,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":18,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 2 items on 2022-01-27 10:54:45!
DEBUG: API Response: {"error":null,"status":1}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":6,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":7,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":8,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":9,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":10,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":11,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":12,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":13,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":14,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":15,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":16,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":17,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":18,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":19,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":20,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":21,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":22,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":23,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
DEBUG: Sent 1 items on 2022-01-27 10:54:45!
DEBUG: API Response: {"code":200,"num_records_imported":1,"status":"OK"}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 10 items on 2022-01-27 10:54:45!
DEBUG: API Response: {"error":null,"status":1}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":6,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":7,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":8,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":9,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":10,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":11,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":12,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":13,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":14,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":15,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":16,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":17,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":18,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":19,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":20,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":21,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":22,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":23,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":24,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":25,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":26,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":27,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":28,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":29,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":30,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":31,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":32,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":33,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
DEBUG: Sent 1 items on 2022-01-27 10:54:46!
DEBUG: API Response: {"code":200,"num_records_imported":1,"status":"OK"}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 23 items on 2022-01-27 10:54:46!
DEBUG: API Response: {"error":null,"status":1}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
WARNING: The server couldn't fulfill the request.
WARNING: Error code: 400
WARNING: Reason: Bad Request
WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":6,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":7,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":8,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":9,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":10,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":11,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":12,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":13,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":14,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":15,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":16,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":17,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":18,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":19,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":20,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":21,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":22,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":23,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":24,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":25,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":26,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":27,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":28,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":29,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":30,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":31,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":32,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":33,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":34,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":35,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":36,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":37,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":38,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":39,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":40,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":41,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":42,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":43,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":44,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'
ERROR: Failed to import batch, dumping to file import_backup.txt
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 2028, in _send_batch
    response = self.request(
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 254, in request
    response = urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
DEBUG: API Response: None
WARNING: API response EMPTY!
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"Content-Type":"application\/json"}
DEBUG: Sent 2 items on 2022-01-27 10:54:46!
DEBUG: API Response: {"code":200,"num_records_imported":2,"status":"OK"}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <masked>"}
DEBUG: Sent 14 items on 2022-01-27 10:54:47!
DEBUG: API Response: 
WARNING: Exception in _async_response_handler_callback!
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1521, in _async_response_handler_callback
    response_data = json.loads(response)
ValueError: Expected object or value
Exception in thread Thread-875:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 592, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 776, in _set
    self._callback(self._value)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1529, in _async_response_handler_callback
    raise e
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1521, in _async_response_handler_callback
    response_data = json.loads(response)
ValueError: Expected object or value
jaredmixpanel commented 2 years ago

@dhpwd Ah, I added the strict=1 parameter (as is recommended) to the /import request, since some of your events don't have $insert_id's they are failing the strict validation...

WARNING: Response: b'{"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":1,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":2,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":3,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":4,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"},{"index":5,"field":"properties.$insert_id","message":"\'properties.$insert_id\' is invalid: must not be missing"}],"num_records_imported":0,"status":"Bad Request"}\n'

See: https://developer.mixpanel.com/reference/import-events#propertiesinsert_id

If you add $insert_id properties that should fix it. I will go ahead and make the strict parameter configurable, but keep the default as True.

jaredmixpanel commented 2 years ago

@dhpwd i noticed that amplitude also has an insert_id property so I will add that to the amplitude_to_mixpanel_map so that it's automatically mapped to our $insert_id property if your amplitude data already has it. if it doesn't and you don't want to add it, you will be able to turn off strict validation.

jaredmixpanel commented 2 years ago

@dhpwd just released v2.2.2 which hopefully fixes this for you. if your amplitude events have an insert_id property you should be good to go. if they do not, you can disable strict_import validation upon initialization...

mputils = MixpanelUtils(
    'ServiceAccountSecretHere',
    token='ProjectTokenHere',
    service_account_username='my-user.12345.mp-service-account',
    project_id=1234567,
        strict_import=False,
)
dhpwd commented 2 years ago

@jaredmixpanel thanks for helping on this, much appreciated

As you pointed out, with the import_from_amplitude method, the library itself downloads the data from Amplitude and prepares the events for import

Re insert ID, yes Amplitude does indeed include it with the exported data although note the key is also prefixed with "$" like Mixpanel so the mapping should be "$insert_id": "$insert_id",

If it doesn't and you don't want to add it, you will be able to turn off strict validation

Not sure how I would add it if it was missing, given the only method being called is import_from_amplitude and the mapping happens under the hood?

Finally after running it again, in addition to still seeing the insert ID errors (due to the above), it finally crashes with the same error as in the original post:

DEBUG: API Response: 
ERROR: Exception in _async_response_handler_callback!
Traceback (most recent call last):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1528, in _async_response_handler_callback
    response_data = json.loads(response)
ValueError: Expected object or value
Exception in thread Thread-350:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 592, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 776, in _set
    self._callback(self._value)
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/mixpanel_utils/__init__.py", line 1528, in _async_response_handler_callback
    response_data = json.loads(response)
ValueError: Expected object or value

So ultimately it's still failing on the error relating to the API response not being an object or value

It should be possible to run the library in editable mode (although I've not done this before), which I'm hoping would allow me to inspect what the invalid value is to help diagnose 🤔

Or let me know if there's anything else I can do to help?

Thanks again,

Dan

jaredmixpanel commented 2 years ago

@dhpwd Well shoot, they list it as insert_id here: https://developers.amplitude.com/docs/http-api-v2#properties-1

Weird, the logs show there is some kind of empty response: DEBUG: API Response:, but I'm not even sure how to get json.loads() to throw ValueError: Expected object or value.

Anyway, added $insert_id to the map and tweaked the async response handler logic a bit more, released as v2.2.3.

Let me know if that fixes it. If not, I'll get an amplitude test project set up and see if I can repro. Sorry for all the trouble!

dhpwd commented 2 years ago

Ok one step closer, thanks

Now seeing the following error:

ERROR: Response: {"code":400,"error":"some data points in the request failed validation","failed_records":[{"index":0,"insert_id":"ajs-next-d4506e5ffc42f66153f31e222b5c98ce:8nDHprEv8m4QQ3shWDV2dH-1","field":"$insert_id","message":"'$insert_id' is invalid: must only contain alphanumeric or hyphen '-' characters"}

This is because Segment was used to send data to Amplitude and their insert IDs look something like "ajs-next-603ad5dc2be409028728edae03107227:8nDHprEv8m4QQ3shWDV2dH-1" – so it's the colon that it doesn't like and, given the prevalence of Segment this will be a common issue

Suggestion: add some regex to filter out characters not allowed e.g.

import re

mp_insert_id = re.sub(r"[^a-zA-Z0-9-]", "", amplitude_insert_id)

This isn't even specific to Segment's format and will just catch anything not permitted by Mixpanel so a good addition to the library imo :)

Dan

jaredmixpanel commented 2 years ago

@dhpwd added that regex here: https://github.com/mixpanel/mixpanel-utils/blob/master/__init__.py#L2295-L2296

released as v2.2.4 let me know how it goes!

dhpwd commented 2 years ago

@jaredmixpanel great, that solved the insert ID issue and the script actually runs to the end without crashing 😉

Now seeing the below pattern consistently in the debug output

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization": "Basic <redacted>"}
DEBUG: Sent 557 items on 2022-02-01 13:39:03!
DEBUG: API Response: {"error":"a temporary failure occurred","status":0}
WARNING: API response NOT OK: {"error":"a temporary failure occurred","status":0}
DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1
DEBUG: Request Headers: {"Authorization": "Basic <redacted>", "Content-Type": "application/json"}
DEBUG: Sent 648 items on 2022-02-01 13:39:03!
DEBUG: API Response: {"code":200,"num_records_imported":648,"status":"OK"}

Not 100% but is it the engage call that's failing? Quick glance at the code and only non-engage calls have content type application/json 🤔

Dan

jaredmixpanel commented 2 years ago

@dhpwd I'm not able to reproduce that error. It says that it was a temporary failure... are you still getting that error? Yes, only /import is using application/json content-type.

I set up a test script that uses the amplitude to mixpanel stuff and both my import and engage calls are succeeding...

$ python3 amp-to-mix.py

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1&project_id=1695321
DEBUG: Request Headers: {"Authorization":"Basic <redacted>","Content-Type":"application\/json"}
DEBUG: Sent 1794 items on 2022-02-01 15:09:14!
DEBUG: API Response: {"code":200,"num_records_imported":1794,"status":"OK"}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/import?strict=1&project_id=1695321
DEBUG: Request Headers: {"Authorization":"Basic <redacted>","Content-Type":"application\/json"}
DEBUG: Sent 160 items on 2022-02-01 15:09:15!
DEBUG: API Response: {"code":200,"num_records_imported":160,"status":"OK"}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <redacted>"}
DEBUG: Sent 986 items on 2022-02-01 15:09:16!
DEBUG: API Response: {"error":null,"status":1}

DEBUG: Request Method: POST
DEBUG: Request URL: https://api-eu.mixpanel.com/engage?verbose=1
DEBUG: Request Headers: {"Authorization":"Basic <redacted>"}
DEBUG: Sent 1 items on 2022-02-01 15:09:20!
DEBUG: API Response: {"error":null,"status":1}
jaredmixpanel commented 2 years ago

@dhpwd any update? or okay to close this issue?

jaredmixpanel commented 2 years ago

Closing for now. Feel free to re-open if issue persists.