"json decoder error" when running `fleetctl apply` - possible client/server version mismatch?

jacobian-latacora commented 9 months ago

When running fleetctl apply with an organization config, I get:

Error: applying custom settings: POST /api/latest/fleet/mdm/profiles/batch received status 400 Bad request: json decoder error

Fleet version:

Client Version:   4.44.0
Server Version:  0.0.0-SNAPSHOT-[7](https://github.com/latacora/fleet-ops/actions/runs/7891795798/job/21536876419?pr=2#step:6:8)9f3b19
Warning: Version mismatch.
Client Version:   4.44.0
Server Version:  0.0.0-SNAPSHOT-79f3b19

NB: this error doesn't happen with fleetctl v 4.43.3

💥 Actual behavior

$ fleetctl apply --dry-run -f config.yaml
Client Version:   4.44.0
Server Version:  0.0.0-SNAPSHOT-[7](https://github.com/latacora/fleet-ops/actions/runs/7891795798/job/21536876419?pr=2#step:6:8)9f3b19
Warning: Version mismatch.
Client Version:   4.44.0
Server Version:  0.0.0-SNAPSHOT-79f3b19
Error: applying custom settings: POST /api/latest/fleet/mdm/profiles/batch received status 400 Bad request: json decoder error

🧑‍💻 Steps to reproduce

See above.

🕯️ More info (optional)

This is a self-hosted Fleet instance, installed via our own Terraform config (which started by referencing yours)
We've worked around this by pinning fleetctl to 4.43.3, which is totally fine for now.

sabrinabuckets commented 9 months ago

Hey @jacobian-latacora thanks for raising this! Are you comfortable sharing the yaml you used that generate this error so we can try to poke at this and repro your issue?

jacobian-latacora commented 9 months ago

Sure! I needed to redact a couple of things but most of it's there:

---
apiVersion: v1
kind: config
spec:
  agent_options:
    config:
      decorators:
        load:
          - SELECT uuid AS host_uuid FROM system_info;
          - SELECT hostname AS hostname FROM system_info;
      options:
        disable_distributed: false
        distributed_interval: 10
        distributed_plugin: tls
        distributed_tls_max_attempts: 3
        logger_tls_endpoint: /api/osquery/log
        logger_tls_period: 10
        pack_delimiter: /
    overrides: { }
  features:
    enable_host_users: true
    enable_software_inventory: true
  fleet_desktop:
    transparency_url: https://fleetdm.com/transparency
  host_expiry_settings:
    host_expiry_enabled: false
    host_expiry_window: 0
  integrations:
    jira: null
    zendesk: null
  mdm:
    apple_bm_default_team: ""
    apple_bm_enabled_and_configured: false
    apple_bm_terms_expired: false
    enable_disk_encryption: false
    enabled_and_configured: false
    end_user_authentication:
      entity_id: ""
      idp_name: ""
      issuer_uri: ""
      metadata: ""
      metadata_url: ""
    macos_migration:
      enable: false
      mode: ""
      webhook_url: ""
    macos_settings:
      custom_settings:
        - "Firewall.mobileconfig"
    macos_setup:
      bootstrap_package: null
      enable_end_user_authentication: false
      macos_setup_assistant: null
    macos_updates:
      deadline: null
      minimum_version: null
    windows_enabled_and_configured: false
  org_info:
    contact_url: https://fleetdm.com/company/contact
    org_logo_url: ""
    org_logo_url_light_background: https://www.latacora.com/images/logo.svg
    org_name: latacora
  server_settings:
    deferred_save_host: false
    enable_analytics: false
    live_query_disabled: false
    query_reports_disabled: false
    server_url: XXXREDACTEDXXX
  smtp_settings:
    authentication_method: authmethod_plain
    authentication_type: authtype_username_password
    configured: false
    domain: ""
    enable_smtp: false
    enable_ssl_tls: true
    enable_start_tls: true
    password: ""
    port: 587
    sender_address: ""
    server: ""
    user_name: ""
    verify_ssl_certs: true
  sso_settings:
    enable_jit_provisioning: false
    enable_jit_role_sync: false
    enable_sso: true
    enable_sso_idp_login: true
    entity_id: XXXREDACTEDXXX
    idp_image_url: ""
    idp_name: XXXREDACTEDXXX
    issuer_uri: ""
    metadata: XXXREDACTEDXXX
    metadata_url: ""
  vulnerability_settings:
    databases_path: ""
  webhook_settings:
    failing_policies_webhook:
      destination_url: ""
      enable_failing_policies_webhook: false
      host_batch_size: 0
      policy_ids: null
    host_status_webhook:
      days_count: 0
      destination_url: ""
      enable_host_status_webhook: false
      host_percentage: 0
    interval: 24h0m0s
    vulnerabilities_webhook:
      destination_url: ""
      enable_vulnerabilities_webhook: false
      host_batch_size: 0

Of note - we discovered this as we were starting to play with MDM policies; you'll see the reference to Firewall.mobileconfig, so it's possible that's related. That points to an unsigned mobileconfig - we ran into a different error using signed configs, but IIRC that's something y'all already knew about and were working on (If that';s not true LMK and I can raise another issue about it)

sabrinabuckets commented 9 months ago

@jacobian-latacora thanks so much for that! I ran a diff on your config vs mine, and changed some settings accordingly, but didn't run into any errors applying the file. I did hit one when I threw in the Firewall.mobileconfig that you have, but that was just a control because I knew it would fail since I do not have that file. I don't know for sure that it would be related, but definitely possible. I can't tell if you're providing a full path to the file, but in any case this is what you should see if that's the cause:

Error: applying fleet config: open Firewall.mobileconfig: no such file or directory

If you haven't yet, I would be curious to hear if you continue to have the same issue after upgrading to the latest patch release, v4.44.1? We included a fix for a bug with uploading batched profiles. And while that technically shouldn't be related to the error you're getting, this is an interesting detail: POST /api/latest/fleet/mdm/profiles/batch.

Would you mind upgrading & trying again? If it's still giving you trouble, I can get this over to engineering to dig into further.

sabrinabuckets commented 9 months ago

@jacobian-latacora just checking back in to see if you've had a chance to upgrade & retest?

sabrinabuckets commented 9 months ago

Going to close this out since we've heard no further updates in ~3 weeks. If you are still experiencing the issue after upgrading to the latest Fleet release (now v4.46.1) please do feel free to re-open this or to submit a new issue.

fleet-release commented 9 months ago

Mismatched versions call, In cloud city, harmony, Fleet's path to resolve.

fleetdm / fleet