opensearch-project / OpenSearch-Dashboards

📊 Open source visualization dashboards for OpenSearch.
https://opensearch.org/docs/latest/dashboards/index/
Apache License 2.0
1.69k stars 885 forks source link

[MD] Logging and Auditing #1986

Open zhongnansu opened 2 years ago

zhongnansu commented 2 years ago

Task breakdown

Research Notes

Some questions we need to answer

Logging

Data source logging will log datasource, query, time, and error, with correct logging setting and client settings in osd.yml

Similar to what we currently have with default single opensearch cluster. It makes use of the event emitter provided by opensearch-js client lib, that hook into internal events, such as request and response. Doc Reference

Current logging

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/5fb4143b11a0a0292ca5cc96391d192a7ef643b3/src/core/server/opensearch/client/configure_client.ts#L51-L72

Auditing

Security Plugin Audit Log feature

[Proposed Solution] OSD Audit Service + Logging service

core - audit service

// @public
export interface AuditableEvent {
    // (undocumented)
    message: string;
    // (undocumented)
    type: string;
}
// @public
export interface Auditor {
    add(event: AuditableEvent): void;
    withAuditScope(name: string): void;
}
// @public
export interface AuditorFactory {
    // (undocumented)
    asScoped(request: OSDRequest): Auditor;
}
// Warning: (ae-missing-release-tag) "AuditTrailSetup" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
//
// @public (undocumented)
export interface AuditTrailSetup {
    register(auditor: AuditorFactory): void;
}

data source plugin -> audit trail client

export class AuditTrailClient implements Auditor {
  private scope?: string;
  constructor(
    private readonly request: OSDRequest,
    private readonly event$: Subject<AuditEvent>,
    private readonly deps: Deps
  ) {}

  public withAuditScope(name: string) {
    if (this.scope !== undefined) {
      throw new Error(`Audit scope is already set to: ${this.scope}`);
    }
    this.scope = name;
  }

  public add(event: AuditableEvent) {
    const user = this.deps.getCurrentUser(this.request);
    // doesn't use getSpace since it's async operation calling ES
    const spaceId = this.deps.getSpaceId ? this.deps.getSpaceId(this.request) : undefined;

    this.event$.next({
      message: event.message,
      type: event.type,
      user: user?.username,
      space: spaceId,
      scope: this.scope,

data source plugin -> plugin.ts

const configSchema = schema.object({
  enabled: schema.boolean({ defaultValue: false }),
  appender: schema.maybe(coreConfig.logging.appenders),
  logger: schema.object({
    enabled: schema.boolean({ defaultValue: false }),
  }),
});

export type AuditTrailConfigType = TypeOf<typeof configSchema>;

export class DataSource implements Plugin {
  private readonly logger: Logger;
  private readonly config$: Observable<AuditTrailConfigType>;
  private readonly event$ = new Subject<AuditEvent>();

  constructor(private readonly context: PluginInitializerContext) {
    this.logger = this.context.logger.get();
    this.config$ = this.context.config.create();
  }

  public setup(core: CoreSetup, deps: DepsSetup) {
    const depsApi = {
      getCurrentUser:
    };

    core.auditTrail.register({
      asScoped: (request: OSDRequest) => {
        return new AuditTrailClient(request, this.event$, depsApi);
      },
    });

    core.logging.configure(
      this.config$.pipe<LoggerContextConfigInput>(
        map((config) => ({
          appenders: {
            auditTrailAppender: this.getAppender(config),
          },
          loggers: [
            {
              // plugins.auditTrail prepended automatically
              context: '',
              level: config.logger.enabled ? 'debug' : 'off',
              appenders: ['auditTrailAppender'],
            },
          ],
        }))
      )
    );
  }
zhongnansu commented 2 years ago

Reference

1. Kibana Audit Service

https://github.com/elastic/kibana/issues/52125 Original PR to initialize Audit service: https://github.com/elastic/kibana/pull/69278

2. Grafana Audit logs for datasource management

Data sources management

Action Distinguishing fields
Create datasource {"action": "create", "resources": [{"type": "datasource"}]}
Update datasource {"action": "update", "resources": [{"type": "datasource"}]}
Delete datasource {"action": "delete", "resources": [{"type": "datasource"}]}
Enable permissions for datasource {"action": "enable-permissions", "resources": [{"type": "datasource"}]}
Disable permissions for datasource {"action": "disable-permissions", "resources": [{"type": "datasource"}]}
Grant datasource permission to role, team, or user {"action": "create", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}*
Remove datasource permission {"action": "delete", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}
Enable caching for datasource {"action": "enable-cache", "resources": [{"type": "datasource"}]}
Disable caching for datasource {"action": "disable-cache", "resources": [{"type": "datasource"}]}
Update datasource caching configuration {"action": "update", "resources": [{"type": "datasource"}]}

* resources may also contain a third item with "type": set to "user" or "team".

3. OpenSearch security plugin audit log example

{
    "audit_cluster_name": "841677925608:new-m6",
    "audit_node_name": "8b168deaa71e1e5da322cc20de7b812b",
    "audit_request_initiating_user": "admin",
    "audit_rest_request_method": "POST",
    "audit_category": "AUTHENTICATED",
    "audit_request_origin": "REST",
    "audit_request_body": "{\"docs\":[{\"_id\":\"visualization:2edf78b0-5395-11e8-99bf-1ba7b1bdaa61\",\"_index\":\".kibana\"}]}",
    "audit_node_id": "xfijqJVHS7SOD9KZVBYdcQ",
    "audit_request_layer": "REST",
    "audit_rest_request_path": "/_mget",
    "@timestamp": "2022-08-03T07:14:18.431+00:00",
    "audit_request_effective_user_is_admin": false,
    "audit_format_version": 4,
    "audit_request_remote_address": "24.19.168.206",
    "audit_rest_request_headers": {
        "Connection": [
            "keep-alive"
        ],
        "User-Agent": [
            "elasticsearch-js/7.10.0-rc.1 (linux 5.4.117-58.216.amzn2.aarch64-arm64; Node.js v10.24.1)"
        ],
        "x-opaque-id": [
            "30c9fa0c-c416-44d8-841a-11e31dccb235"
        ],
        "Host": [
            "localhost:9200"
        ],
        "x-opensearch-product-origin": [
            "opensearch-dashboards"
        ],
        "Content-Length": [
            "90"
        ],
        "Content-Type": [
            "application/json"
        ]
    },
    "audit_request_effective_user": "admin"
}
zhongnansu commented 2 years ago

@zengyan-amazon @seraphjiang Any thoughts?