Allow to obfuscate body

[Klapsa2503]

Add ability to obfuscate json, xml, form data properties in bodies

Detailed Description

In addition to headers, parameters, paths that can be now configured for obfuscation. I would like to add additional configs to obfuscate json, xml, form data parameters in body. Given for example:

<data>
  <param1>random data</param1>
  <sensitiveParam>secret</sensitiveParam>
</data>

when configured

logbook:
  obfuscate:
    body-xml-property:
      - "sensitiveParam"

should be obfuscated to

<data>
  <param1>random data</param1>
  <sensitiveParam>XXX</sensitiveParam>
</data>

Context

In requests/responses we log we very often transmit sensitive data that should be hidden. This config would make it far easier to obfuscate bodies.

Possible Implementation

• Add config parameters

  logbook:
  obfuscate:
  body-xml-property:
    - "smth1"
  body-json-property:
    - "smth2"
  body-form-data-property:
    - "smth3"

• Configure by default in LogbookAutoConfiguration.java
• Implement property matchers using regex similar to QueryFilters::replaceQuery
• Given matcher should only be applied for certain content types

Your Environment

• Version used: 2.14.0

I will be happy to contribute to the project if only this gets approval

[Klapsa2503]

Json body filter JacksonJsonFieldBodyFilter and BodyFilters::replaceFormUrlEncodedProperty could be used for json and form data. Additionally https://stackoverflow.com/questions/65718548/how-to-mask-sensitive-data-in-a-xml-body-with-zalando-logbook/74109250 for xml

[SpiReCZ]

We made our own implementation for JSON filtering on top of the already provided one. It uses JsonPath and filters requests/responses only for specific URLs (APIs). Data is replaced only when not null to not change context of the message. Each of our microservice has it's own set of filters specific to used APIs, it is applicable for both HTTP server/client.

The whole implementation is a bit longer than the example bellow provided and for Path matching of HTTP responses we needed to store the original URL inside HTTP response's HTTP headers. It is not ideal solution, but works perfectly. I hope that logbook would some day support to pass some "context" between request/response like the request URL.

request/response filter example:

package com.example.config;

import com.example.logging.filter.BodyFilterConstants;
import com.example.logging.filter.PathMatchingBodyFilteredHttpRequest;
import com.example.logging.filter.PathMatchingBodyFilteredHttpResponse;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.zalando.logbook.BodyFilter;
import org.zalando.logbook.RequestFilter;
import org.zalando.logbook.ResponseFilter;

import java.util.stream.Stream;

import static com.example.logging.CustomConditions.responseTo;
import static com.example.logging.filter.JsonPathSafeBodyFilters.jsonPath;
import static org.zalando.logbook.Conditions.requestTo;

@Configuration
@ConditionalOnProperty(name = "app.logging.http.obfuscate-json", matchIfMissing = true)
public class LogbookObfuscationConfig {
  @Bean
  public RequestFilter eventEndpointRequestFilter() {
    BodyFilter requestJsonFilter = Stream.of(
            jsonPath("$.username").replace(BodyFilterConstants.OBFUSCATED_STRING_OR_PASSWORD),
            jsonPath("$.password").replace(BodyFilterConstants.OBFUSCATED_STRING_OR_PASSWORD)
        )
        .reduce(BodyFilter::merge)
        .orElse(null);
    return request -> new PathMatchingBodyFilteredHttpRequest(request, requestTo("**/some-events"), requestJsonFilter);
  }
  @Bean
  public ResponseFilter eventsFilter() {
    return response -> new PathMatchingBodyFilteredHttpResponse(
        response,
        responseTo("**/some-events/*"),
        jsonPath("$.eventDetail.sensitive-data").replace(BodyFilterConstants.OBFUSCATED_SENSITIVE_DATA_EXAMPLE)
    );
  }
}

[whiskeysierra]

No objections from my side for an XML body filter. I just never had the need for it, since I just happen to use JSON exclusively (not 100% my choice, so one could say I got lucky).

I'd separate that from the configuration-based approach though. The way I see it, there are some aspects which are better suited to configuration than others. URL exclusion/inclusion filters, query/header replacement, etc. is all fine. But configuring a chain of body filters in the right sequence, for different content types, using different replacement patterns, etc. feels a bit much and is probably something better done in code.

[bata19]

Hi, I have made a private project on top of logbook (happy to open source or add PR to add support into this project) that does:

1. JSON and XML obfuscation, base on config. Especially nice with Spring Boot, can set details for every field in application.yaml file.
2. Creates some modules to add additional logbook to simultaneously save to database (we were allowed to save non obfuscated req/res to database to improve tracing, and logs in prod after certain point should only log errors)

Look at the README from that project.

How to mask data with Logbook

• for mvc and restTemplate settings import log-masking-spring-boot-starter. It also imports org.zalando:logbook-spring-boot-starter with lots of default settings

  dependencies {
  implementation("com.github.bata19:log-masking-spring-boot-starter:VERSION")
  }

If you define your own restTemplate it needs to be defined using builder. Customizer that it is used to add logging interceptor is using RestTemplateBuilder

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder builder) {
        return builder.build();
    }

• for webflux and webClient using netty import

  dependencies {
  implementation("com.github.bata19:log-masking-spring-boot-starter:VERSION")
  implementation("org.zalando:logbook-spring-boot-webflux-autoconfigure:VERSION")
  implementation("org.zalando:logbook-netty:VERSION")
  }

  For WebClient you need to add:

    @Bean
    public WebClient webClient(final Logbook logbook) {
        return WebClient.builder()
                .filter(new LogbookExchangeFilterFunction(logbook))
                .build();
    }

In the application.yaml file:

logbook.masking.enabled: true/false enables the masking on or off

Set masking strategy:

logbook.replace.strategy

• blacklist - list fields you want to mask and how (default)
• whitelist - list fields you don't want to mask and applies default masking to unlisted fields
• whitelistExclusive - list fields you don't want to mask and masks unlisted fields
  

Obfuscate Headers

logbook.obfuscate.headers masks specific headers

Obfuscate JSON fields

logbook.obfuscate.jsonElement mask specific JSON fields

Obfuscate XML Elements

logbook.obfuscate.xmlElement mask child nodes only

Obfuscate formUrlEncodedContent

logbook.obfuscate.formUrlEncodedContent

Describe every field desired masking

If any element for specific field is not set default setting will be used. You can override default setting in logbook.obfuscate.defaultValues field.

• replace.mode Number of elements to leave (l, leave) or to replace (r, replace) in the string
• replace.position Start masking elements from head (h, head) or tail (t, tail)
• replace.length Number of characters we want to mask
• format.value Replacement value as a string (any non empty string)
• format.type Replace every character with string (st, string) or number fo caharacters with constant (co, constant)

Example

  obfuscate:
    defaultValues:
      replace:
        mode: r
        position: h
        length: 2
      format:
        type: st # as
        value: "#" # with
    headers:
      #            - Authorization
      #            - X-Secret
      - random # so we can print all headers
    parameters:
      - access_token
      - password
    jsonElement:
      accountNumber:
      originalAmount:
        replace:
          position: t
          length: 4
        format:
          value: "*" # with
      alternateKey:
        replace:
          length: 1000
      stan:
        format:
#          type: const
          value: "@"
      description:
        format:
          value: $
        replace:
          position: t
          length: 1000
    xmlElement:
      MsgFctn:
      Id:
        format:
          value: "*"
        replace:
          position: t
          length: 4
      PANToken:
        replace:
          length: 1000
      InitrTxId:
        format:
          value: "@"
      Othr:
        replace:
          position: t
          length: 1000
        format:
          value: $
      TxLifeCyclId:
        replace:
          mode: l
          length: 7
          position: h
        format:
          value: $
          type: string
      MrchntCtgyCd:
        replace:
          mode: l
          position: t
        format:
          value: ^
          type: const
      CstmrNb:
        replace:
          mode: l
          length: 1000
    write:
      chunk-size: 1000
      category: http.wire-log

How to add logging to database

Adding support in springboot

• To use this in spring boot app you need to include log-masking-database-spring-boot-starter. And implement interface DatabaseHttpLogWriter.

• To add default database implementation for DatabaseHttpLogWriter, include log-masking-database-hibernate-spring-boot-starter into your app and set datasource in application.yaml.

Example build.gradle.kts file

plugins {
    java
    id("org.springframework.boot") version "2.7.8"
    id("io.spring.dependency-management") version "1.0.15.RELEASE"
}

group = "com.brnslv.mask"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17

configurations {
    compileOnly {
        extendsFrom(configurations.annotationProcessor.get())
    }
}

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation("com.github.bata19:log-masking-spring-boot-starter:VERSION")
    implementation("com.github.bata19:log-masking-database-spring-boot-starter:VERSION")
    implementation("com.github.bata19:log-masking-database-hibernate-spring-boot-starter:VERSION")
    implementation("org.springframework.boot:spring-boot-starter-data-jpa")
    implementation("org.springframework.boot:spring-boot-starter-web")
    compileOnly("org.projectlombok:lombok")
    runtimeOnly("org.postgresql:postgresql")
    annotationProcessor("org.projectlombok:lombok")
    testImplementation("org.springframework.boot:spring-boot-starter-test")
}

tasks.withType<Test> {
    useJUnitPlatform()
}

Turn on/off database logging and logger logging:

logbook.write.database

• true(default)
• false

logbook.write.logger

• true(default)
• false

If all are set to false, default Logbook settings from org.zalando:logbook-spring-boot-starter is used.

[github-actions[bot]]

In order to prioritize the support for Logbook, we would like to check whether the old issues are still relevant. This issue has not been updated for over six months.

• Please check if it is still relevant in latest version of the Logbook.
• If so, please add a descriptive comment to keep the issue open.
• Otherwise, the issue will automatically be closed after a week.

[Klapsa2503]

This case is still valid, I just didn't have the time to implement it. I will try to do it next month.

[Klapsa2503]

No time to implement this

[SpiReCZ]

@Klapsa2503 to respond on how i made it possible to filter Json responses by path... I had to hack here and there a little, since otherwise I would have to rewrite Logbook core classes. I just inject the path from the request to the response headers. Response class is custom made to get the path from header. Then I make sure to remove that header at the end in Http Response when finalizing operation toString()/writeBodyText or something similar is called.

[whiskeysierra]

A custom strategy should work for you without the need to hack anything.

On Mon, Feb 5, 2024, 00:14 SpiReCZ @.***> wrote:

@Klapsa2503 https://github.com/Klapsa2503 to respond on how i made it possible to filter Json responses by path... I had to hack here and there a little, since otherwise I would have to rewrite Logbook core classes. I just inject the path from the request to the response headers. Response class is custom made to get the path from header. Then I make sure to remove that header at the end in Http Response when finalizing operation toString()/writeBodyText or something similar is called.

— Reply to this email directly, view it on GitHub https://github.com/zalando/logbook/issues/1366#issuecomment-1925962380, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADI7HK7QTJ4FL2JLDC6N5TYSAI6PAVCNFSM6AAAAAARG3HH7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVHE3DEMZYGA . You are receiving this because you commented.Message ID: @.***>