search

elastic / apm-agent-python

https://www.elastic.co/guide/en/apm/agent/python/current/index.html
BSD 3-Clause "New" or "Revised" License
414 stars 220 forks source link

capture_exception raise TransportException and abort transaction #1440

Open luipir opened 2 years ago

luipir commented 2 years ago

Describe the bug: ...

During an exception handling capture_exeption fail to send data and as result transaction is aborted. Seems the problem is decoding the data payload (memoryview type) to urllib3

This snippet replicate the issue in my env, just need to setup elasticapm client:

from shapely.geometry import Polygon
import geopandas as gpd
import elasticapm
import traceback

client = elasticapm.Client(<a config dictionary>)
client.begin_transaction(transaction_type="script")
result = 'unknown'
try:
    with elasticapm.capture_span(name="a_name", labels={}):
        a = Polygon()
        b = gpd.GeoDataFrame(geometry=[a])
        b.centroid.x[0]
    result = 'success'
except Exception:
    message = traceback.format_exc()
    client.capture_exception()
    result = 'failed'
finally:
    client.end_transaction(name="transaction_name", result=result)

result in the following trace:

2021-12-22 12:22:56,737 [urllib3.connectionpool] [DEBUG] https://<hide>:443 "POST /intake/v2/events HTTP/1.1" 400 None
2021-12-22 12:22:56,828 [elasticapm.transport] [ERROR] Failed to submit message: 'HTTP 400: {"accepted":24,"errors":[{"message":"decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Stacktrace: []v2.stacktraceFrame: v2.stacktraceFrame.Vars: Read: unexpected value type: 0, error found in #10 byte of ...|l_value\\": NaN}, \\"pre|..., bigger context ...|,\\\\n      dtype=object)\\", \\"op\\": \\"x\\", \\"null_value\\": NaN}, \\"pre_context\\": [\\"    # type: (str, np.array[|...","document":"{\\"error\\": {\\"context\\": {\\"custom\\": {}}, \\"id\\": \\".....\\", \\"exception\\": {\\"message\\": \\"IndexError: list index out of range\\", \\"type\\": \\"IndexError\\", \\"module\\": \\"builtins\\", \\"stacktrace\\": [{\\"abs_path\\": \\"./stripe_delivery_pipeline.py\\", \\"filename\\": \\"stripe_delivery_pipeline.py\\", \\"module\\": \\"__main__\\", \\"function\\": \\"main\\", \\"lineno\\": 1970, \\"library_frame\\": false, \\"vars\\": {\\"elasticapm_span_labels\\": null, \\"elasticapm_span_name\\": \\"delivery-pipeline-main\\", \\"location\\": \\"undefined\\", \\"country\\": \\"undefined\\", \\"message\\": \\"Traceback (most recent call last):\\\\n  File \\\\\\"./stripe_delivery_pipeline.py\\\\\\", line 1970, in main\\\\n    status, message, country, location = real_main_multi_AOI(scene_set_id, scene_id_tag)\\\\n  File \\\\\\"./stri...\\", \\"status\\": \\"Failed\\", \\"result\\": null, \\"labels\\": {\\"pod_name\\": null, \\"job_id\\": null, \\"sceneset_id\\": \\".........\\", \\"scene_id_tag\\": null}, \\"     ...[cut]

and

Traceback (most recent call last):
  File "/home/lpirelli/miniconda/envs/py36/lib/python3.6/site-packages/elasticapm/transport/base.py", line 239, in _flush
    self.send(data)
  File "/home/lpirelli/miniconda/envs/py36/lib/python3.6/site-packages/elasticapm/transport/http.py", line 113, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.exceptions.TransportException: HTTP 400: {"accepted":24,"errors":[{"message":"decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Stacktrace: []v2.stacktraceFrame: v2.stacktraceFrame.Vars: Read: unexpected value type: 0, error found in #10 byte of ...|l_value\": NaN}, \"pre|..., bigger context ...|,\\n      dtype=object)\", \"op\": \"x\", \"null_value\": NaN}, \"pre_context\": [\"    # type: (str, np.array[|...","document":"{\"error\": {\"context\": {\"custom\": {}}, \"id\": \"....\", \"exception\": {\"message\": \"IndexError: list index out of range\", \"type\": \"IndexError\", \"module\": \"builtins\", \"stacktra    ....[cut]

data payload in memory view is:

a= b'\x1f\x8b\x08\x00a\x11\xc3a\x02\xff\xed}{w#\xb7\xb1\xe7\xff\xfb)xx\xb3\xeb\xf1\x1e\x91\xc2\xb3\x1f:q\xb2\xb2F\xb6\x95;\x0fE\x92\x938\xe39}\xd0\xddh\xa93$\x9b\xe9&53\xf1z?\xfbV\x01\xfd\xe4C\xa44\x92F\xf6pr\xaf%u\x03\xe8z\xa1\xf0.... [cut]'

str(a, 'utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Environment (please complete the following information)

luipir commented 2 years ago

this is my conda list:

    _libgcc_mutex             0.1                        main    conda-forge
    affine                    2.3.0                      py_0    conda-forge
    appdirs                   1.4.4                    pypi_0    pypi
    argon2-cffi               20.1.0           py36h1d69622_2    conda-forge
    argparse                  1.4.0                    pypi_0    pypi
    astroid                   2.4.2                    py36_0  
    async_generator           1.10                       py_0    conda-forge
    attrs                     20.3.0             pyhd3eb1b0_0  
    autopep8                  1.5.4                      py_0  
    backcall                  0.2.0                      py_0  
    beautifulsoup4            4.10.0                   pypi_0    pypi
    black                     20.8b1                   pypi_0    pypi
    blas                      1.0                         mkl    conda-forge
    bleach                    3.2.1              pyh9f0ad1d_0    conda-forge
    boto3                     1.16.28                  pypi_0    pypi
    botocore                  1.19.28                  pypi_0    pypi
    bzip2                     1.0.8                h7b6447c_0  
    ca-certificates           2021.1.19            h06a4308_0  
    cached-property           1.5.2                    pypi_0    pypi
    cachetools                4.1.1                    pypi_0    pypi
    cairo                     1.14.12              h8948797_3  
    certifi                   2020.12.5        py36h06a4308_0  
    cffi                      1.14.4           py36h211aa47_0    conda-forge
    cfitsio                   3.470                hb7c8383_2  
    chardet                   3.0.4                    pypi_0    pypi
    click                     7.1.2                      py_0  
    click-plugins             1.1.1                      py_0    conda-forge
    cligj                     0.7.1            py36h06a4308_0  
    confluent-kafka           1.7.0                    pypi_0    pypi
    coreapi                   2.3.4+v20180822          pypi_0    pypi
    coreschema                0.0.4                    pypi_0    pypi
    curl                      7.67.0               hbc83047_0  
    cycler                    0.10.0                   py36_0    conda-forge
    dataclasses               0.8                      pypi_0    pypi
    dbus                      1.13.18              hb2f20db_0  
    decorator                 4.4.2                      py_0    conda-forge
    defusedxml                0.6.0                      py_0    conda-forge
    descartes                 1.1.0                    pypi_0    pypi
    df2gspread                1.0.4                    pypi_0    pypi
    dicttoxml                 1.7.4                    pypi_0    pypi
    elastic-apm               6.4.0                    pypi_0    pypi
    elasticsearch             7.10.1                   pypi_0    pypi
    entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
    expat                     2.2.10               he6710b0_2  
    fiona                     1.8.13.post1     py36hc820daa_0  
    fontconfig                2.13.0               h9420a91_0  
    freetype                  2.10.4               h5ab3b9f_0  
    freexl                    1.0.5                h14c3975_0  
    freezegun                 1.0.0                      py_0    conda-forge
    furl                      2.1.0                    pypi_0    pypi
    gdal                      3.0.2            py36hbb2a789_0  
    geographiclib             1.50                     pypi_0    pypi
    geojson                   2.5.0                    pypi_0    pypi
    geopandas                 0.8.1                      py_0    conda-forge
    geopy                     2.1.0                    pypi_0    pypi
    geos                      3.8.0                he6710b0_0  
    geotiff                   1.5.1                h21e8280_1  
    giflib                    5.1.4                h14c3975_1  
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.18                   pypi_0    pypi
    glib                      2.63.1               h5a9c865_0  
    google                    3.0.0                    pypi_0    pypi
    google-api-python-client  1.6.7                    pypi_0    pypi
    google-auth               1.23.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.2                    pypi_0    pypi
    gspread                   3.6.0                    pypi_0    pypi
    gst-plugins-base          1.14.0               hbbd80ab_1  
    gstreamer                 1.14.0               hb453b48_1  
    h5py                      3.1.0                    pypi_0    pypi
    hdf4                      4.2.13               h3ca952b_2  
    hdf5                      1.10.4               hb1b8bf9_0  
    html5lib                  1.1                      pypi_0    pypi
    httplib2                  0.18.1                   pypi_0    pypi
    icu                       58.2                 he6710b0_3  
    idna                      2.10                     pypi_0    pypi
    imageio                   2.9.0                    pypi_0    pypi
    importlib-metadata        3.3.0            py36h5fab9bb_2    conda-forge
    importlib_metadata        3.3.0                hd8ed1ab_2    conda-forge
    iniconfig                 1.1.1                      py_0  
    intel-openmp              2020.2                      254  
    ipdb                      0.13.9                   pypi_0    pypi
    ipykernel                 5.5.5            py36hcb3619a_0    conda-forge
    ipython                   7.16.1           py36h5ca1d4c_0  
    ipython_genutils          0.2.0              pyhd3eb1b0_1  
    isort                     5.6.4                      py_0    conda-forge
    itypes                    1.2.0                    pypi_0    pypi
    jedi                      0.17.2           py36h06a4308_1  
    jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
    jmespath                  0.10.0                   pypi_0    pypi
    joblib                    1.1.0                    pypi_0    pypi
    jpeg                      9b                   h024ee3a_2  
    json-c                    0.13.1               h1bed415_0  
    json2xml                  3.6.0                    pypi_0    pypi
    jsonschema                3.2.0                      py_2    conda-forge
    jupyter_client            6.1.7                      py_0    conda-forge
    jupyter_core              4.7.0            py36h06a4308_0  
    jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
    kealib                    1.4.7                hd0c454d_6  
    kiwisolver                1.3.0            py36h2531618_0  
    krb5                      1.16.4               h173b8e3_0  
    lazy-object-proxy         1.4.3            py36h7b6447c_0  
    lcms2                     2.11                 h396b838_0  
    ld_impl_linux-64          2.33.1               h53a641e_7    conda-forge
    libboost                  1.67.0               h46d08c1_4  
    libcurl                   7.67.0               h20c2e04_0  
    libdap4                   3.19.1               h6ec2957_0  
    libedit                   3.1.20191231         h14c3975_1  
    libffi                    3.2.1             hf484d3e_1007  
    libgcc-ng                 9.1.0                hdf63c60_0  
    libgdal                   3.0.2                h27ab9cc_0  
    libgfortran-ng            7.3.0                hdf63c60_0  
    libkml                    1.3.0                h590aaf7_4  
    libnetcdf                 4.6.1                h11d0813_2  
    libpng                    1.6.37               hbc83047_0  
    libpq                     11.2                 h20c2e04_0  
    libsodium                 1.0.18               h7b6447c_0  
    libspatialindex           1.9.3                he6710b0_0  
    libspatialite             4.3.0a               h793db0d_0  
    libssh2                   1.9.0                h1ba5d50_1  
    libstdcxx-ng              9.1.0                hdf63c60_0  
    libtiff                   4.1.0                h2733197_0  
    libuuid                   1.0.3                h1bed415_2  
    libxcb                    1.14                 h7b6447c_0  
    libxml2                   2.9.10               hb55368b_3  
    lxml                      4.6.3                    pypi_0    pypi
    lz4-c                     1.8.1.2              h14c3975_0  
    markupsafe                1.1.1            py36he6145b8_2    conda-forge
    matplotlib                3.3.2                         0    conda-forge
    matplotlib-base           3.3.2            py36h817c723_0  
    mccabe                    0.6.1                    py36_1  
    mistune                   0.8.4           py36h1d69622_1002    conda-forge
    mkl                       2020.2                      256  
    mkl-service               2.3.0            py36he904b0f_0  
    mkl_fft                   1.2.0            py36h23d657b_0  
    mkl_random                1.1.1            py36h0573a6f_0  
    more-itertools            8.6.0              pyhd3eb1b0_0  
    munch                     2.5.0                      py_0    conda-forge
    mypy_extensions           0.4.3                    py36_0    conda-forge
    nbclient                  0.5.1                      py_0    conda-forge
    nbconvert                 6.0.7            py36h5fab9bb_3    conda-forge
    nbformat                  5.0.8                      py_0    conda-forge
    ncurses                   6.2                  he6710b0_1  
    nest-asyncio              1.4.3              pyhd8ed1ab_0    conda-forge
    networkx                  2.5.1                    pypi_0    pypi
    notebook                  6.1.5            py36h5fab9bb_0    conda-forge
    numpy                     1.19.2           py36h54aff64_0  
    numpy-base                1.19.2           py36hfa32c7d_0  
    oauth2client              4.1.3                    pypi_0    pypi
    oauthlib                  3.1.0                    pypi_0    pypi
    olefile                   0.46                     py36_0  
    opencv-python             4.4.0.46                 pypi_0    pypi
    opencv-python-headless    4.5.4.58                 pypi_0    pypi
    openjpeg                  2.3.0                h05c96fa_1  
    openssl                   1.1.1i               h27cfd23_0  
    orderedmultidict          1.0.1                    pypi_0    pypi
    packaging                 20.8               pyhd3deb0d_0    conda-forge
    pandas                    1.1.3            py36he6710b0_0  
    pandoc                    2.11.2               h36c2ea0_0    conda-forge
    pandocfilters             1.4.2                      py_1    conda-forge
    parso                     0.7.0                      py_0  
    pathspec                  0.8.1                    pypi_0    pypi
    pcre                      8.44                 he6710b0_0  
    pexpect                   4.8.0              pyhd3eb1b0_3  
    pickleshare               0.7.5           pyhd3eb1b0_1003  
    pillow                    8.0.1            py36he98fc37_0  
    pip                       20.3             py36h06a4308_0  
    pixman                    0.40.0               h7b6447c_0  
    pluggy                    0.13.1                   py36_0  
    poppler                   0.65.0               h581218d_1  
    poppler-data              0.4.10               h06a4308_0  
    postgresql                11.2                 h20c2e04_0  
    proj                      6.2.1                haa6030c_0  
    prometheus_client         0.9.0              pyhd3deb0d_0    conda-forge
    prompt-toolkit            3.0.8                      py_0    conda-forge
    ptyprocess                0.6.0              pyhd3eb1b0_2  
    py                        1.10.0             pyhd3eb1b0_0  
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycodestyle               2.6.0                      py_0  
    pycparser                 2.20               pyh9f0ad1d_2    conda-forge
    pygments                  2.7.2              pyhd3eb1b0_0  
    pylint                    2.6.0                    py36_0  
    pyparsing                 2.4.7                      py_0  
    pyproj                    2.6.1.post1      py36hd003209_1  
    pyqt                      5.9.2            py36h05f1152_2  
    pyrsistent                0.17.3           py36h1d69622_1    conda-forge
    pytest                    6.2.4                    pypi_0    pypi
    pytest-mock               3.5.0                    pypi_0    pypi
    python                    3.6.10               h191fe78_1  
    python-dateutil           2.8.1                      py_0    conda-forge
    python-dotenv             0.15.0                   pypi_0    pypi
    python_abi                3.6                     1_cp36m    conda-forge
    pytz                      2020.4             pyhd3eb1b0_0  
    pywavelets                1.1.1                    pypi_0    pypi
    pyzmq                     20.0.0           py36h2531618_1  
    qt                        5.9.7                h5867ecd_1  
    rasterio                  1.1.0            py36h41e4f33_0  
    rasterstats               0.15.0                   pypi_0    pypi
    readline                  7.0                  h7b6447c_5  
    regex                     2020.11.13       py36h27cfd23_0  
    requests                  2.25.0                   pypi_0    pypi
    requests-oauthlib         1.3.0                    pypi_0    pypi
    requests-toolbelt         0.9.1                    pypi_0    pypi
    retry                     0.9.2                    pypi_0    pypi
    rsa                       4.6                      pypi_0    pypi
    rtree                     0.9.4                    py36_1  
    s3transfer                0.3.3                    pypi_0    pypi
    scikit-image              0.17.2                   pypi_0    pypi
    scikit-learn              0.24.2                   pypi_0    pypi
    scipy                     1.5.4                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.5.0                      py_0    conda-forge
    setuptools                50.3.2           py36h06a4308_2  
    shapely                   1.7.1            py36h98ec03d_0  
    simplejson                3.17.3                   pypi_0    pypi
    sip                       4.19.8           py36hf484d3e_0  
    six                       1.15.0           py36h06a4308_0  
    slacker                   0.14.0                   pypi_0    pypi
    smmap                     5.0.0                    pypi_0    pypi
    snuggs                    1.4.7                      py_0    conda-forge
    soupsieve                 2.2.1                    pypi_0    pypi
    sqlite                    3.33.0               h62c20be_0  
    tbb                       2018.0.5             h6bb024c_0  
    terminado                 0.9.1            py36h5fab9bb_1    conda-forge
    testpath                  0.4.4                      py_0    conda-forge
    threadpoolctl             3.0.0                    pypi_0    pypi
    tifffile                  2020.9.3                 pypi_0    pypi
    tiledb                    1.6.3                h1fb8f14_0  
    tk                        8.6.10               hbc83047_0  
    toml                      0.10.2                   pypi_0    pypi
    torch                     1.7.1                    pypi_0    pypi
    tornado                   6.1              py36h27cfd23_0  
    tqdm                      4.54.0                   pypi_0    pypi
    traitlets                 4.3.3                    py36_0    conda-forge
    typed-ast                 1.4.1            py36h7b6447c_0  
    typing_extensions         3.7.4.3                    py_0    conda-forge
    uritemplate               3.0.1                    pypi_0    pypi
    urllib3                   1.26.2                   pypi_0    pypi
    utm                       0.7.0                    pypi_0    pypi
    wcwidth                   0.2.5                      py_0  
    webencodings              0.5.1                      py_1    conda-forge
    wheel                     0.36.0             pyhd3eb1b0_0  
    wrapt                     1.11.2           py36h7b6447c_0  
    xerces-c                  3.2.3                h780794e_0  
    xmltodict                 0.11.0                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0  
    zeromq                    4.3.3                he6710b0_3  
    zipp                      3.4.0                      py_0    conda-forge
    zlib                      1.2.11               h7b6447c_3  
    zstd                      1.3.7                h0b5b093_0
basepi commented 2 years ago

I haven't had a chance to test this yet, but can you define what you mean by "and as result transaction is aborted."? If urllib3 is indeed throwing an error because of a payload it can't decode correctly, that's one thing. But when the transport raises an error, it should be logged but I think the transaction should still be reported.

luipir commented 2 years ago

Hi @basepi sorry for my delay... just come back to work. Need some time to come back on this issue. I've give you a feedback as soon as possible

basepi commented 2 years ago

@luipir Any luck finding some time here?

luipir commented 2 years ago

not yet sorry :/

rwikdutta commented 1 year ago

Hi, Any update on this? This is affecting me as well

basepi commented 1 year ago

@rwikdutta Can you share any stacktraces and additional information? Want to make sure I have a clear idea of the behavior.