`field type conflict: input field \"connect_time\" on measurement \"nginx_access_logs\" is type string, already exists as type float`

[raldone01]

I sucessfully setup the maxminddb download and the nginx log format.

However when using influxdbv2 it fails to write data because of a type mismatch.

<redacted for privacy>
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json; charset=utf-8', 'X-Influxdb-Build': 'OSS', 'X-Influxdb-Version': 'v2.7.10', 'X-Platform-Error-Code': 'unprocessable entity', 'Date': 'Wed, 04 Sep 2024 08:07:23 GMT', 'Content-Length': '236'})
HTTP response body: {"code":"unprocessable entity","message":"failure writing points to database: partial write: field type conflict: input field \"connect_time\" on measurement \"nginx_access_logs\" is type string, already exists as type float dropped=1"}

In the redacted part I spotted connect_time=-\\ which may be causing the issue. There is one request with no connect time. The others have it correctly set to a float.

It would also be nice to describe the required permissions for the influxv2 access token. Currently just giving access to a single bucket is not enough as the script needs to write to the organisation (I don't know what it writes).

[GilbN]

Can you share the log line?

[raldone01]

I don't want this line on a public issue tracker. It is 3605 chars long and I don't want to redact it all manually. Can I send it to you via email or share it in some other way?

[GilbN]

@raldone01 should be fixed on latest. Changed the minium access level for the token. Just needs write access to the bucket now.

[raldone01]

Only write permissions work now.

nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | InfluxClient | INFO     | (influxv2.create_org|line:184) | Creating organization. |
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | InfluxClient | ERROR    | (influxv2.create_org|line:189) | Error creating organization main. |'Traceback (most recent call last):\n  File "/config/geoip2db/geoip2influx/influxv2.py", line 185, in create_org\n    self.org_api.create_organization(name=self.org)\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/client/organizations_api.py", line 45, in create_organization\n    return self._organizations_service.post_orgs(post_organization_request=organization)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/service/organizations_service.py", line 1062, in post_orgs\n    (data) = self.post_orgs_with_http_info(post_organization_request, **kwargs)  # noqa: E501\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/service/organizations_service.py", line 1084, in post_orgs_with_http_info\n    return self.api_client.call_api(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/_sync/api_client.py", line 343, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/_sync/api_client.py", line 173, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/_sync/api_client.py", line 388, in request\n    return self.rest_client.POST(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/_sync/rest.py", line 311, in POST\n    return self.request("POST", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/_sync/rest.py", line 261, in request\n    raise ApiException(http_resp=r)\ninfluxdb_client.rest.ApiException: (401)\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Content-Type\': \'application/json; charset=utf-8\', \'X-Influxdb-Build\': \'OSS\', \'X-Influxdb-Version\': \'v2.7.10\', \'X-Platform-Error-Code\': \'unauthorized\', \'Date\': \'Tue, 10 Sep 2024 15:57:08 GMT\', \'Content-Length\': \'69\'})\nHTTP response body: {\n\t"code": "unauthorized",\n\t"message": "write:orgs is unauthorized"\n}\n'|
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | InfluxClient | INFO     | (influxv2.create_bucket|line:158) | Creating bucket. |
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | InfluxClient | ERROR    | (influxv2.create_bucket|line:165) | Error creating bucket feather-ink-nginx-geoip2influx. |'Traceback (most recent call last):\n  File "/config/geoip2db/geoip2influx/influxv2.py", line 161, in create_bucket\n    self.bucket_api.create_bucket(bucket_name=self.bucket, org=self.org, description=bucket_description, retention_rules=bucket_retention)\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/client/bucket_api.py", line 56, in create_bucket\n    org_id=get_org_query_param(org=(org_id if org is None else org),\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/lsiopy/lib/python3.12/site-packages/influxdb_client/client/util/helpers.py", line 40, in get_org_query_param\n    raise InfluxDBError(response=None, message=message)\ninfluxdb_client.client.exceptions.InfluxDBError: The client cannot find organization with name: \'main\' to determine their ID. Are you using token with sufficient permission?'|
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | InfluxClient | SUCCESS  | (influxv2.setup|line:115) | InfluxDB client setup complete. |
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | LogParser    | INFO     | (logparser.log_file_exists|line:120) | Log file /config/log/nginx/access_geoip2.log exists. |
nginx-geoip2influx-1  | 10/09/2024 17:57:08 | MainThread        | LogParser    | INFO     | (logparser.geoip_file_exists|line:131) | GeoIP file /config/geoip2db/GeoLite2-City.mmdb exists. |
nginx-geoip2influx-1  | 10/09/2024 17:57:10 | MainThread        | LogParser    | SUCCESS  | (logparser.validate_log_format|line:108) | Log file format is valid! |
nginx-geoip2influx-1  | 10/09/2024 17:57:10 | MainThread        | LogParser    | INFO     | (logparser.tail_logs|line:153) | Tailing log file. |

Maybe you could catch the exception and log something like:

Skipping org creation because of insufficient permissions.
Skipping bucket creation because of insufficient permissions.

I have just pulled in the latest image and so far the error has not occured. I will open the issue again if it happens again and mail you the log line.

Thanks for your work.

[GilbN]

Yeah, good idea. I've pushed a commit that catches the permission errors