minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

Add command for uploading file to SUBNET issue #4854

Closed anjalshireesh closed 8 months ago

anjalshireesh commented 8 months ago

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers under the terms of the Apache 2 license. By creating this pull request I represent that I have the right to license the contributions to the project maintainers under the Apache 2 license.

Description

mc support upload --issue <issueNum> --comment <msg> <alias> </path/to/file>

Will upload the given file to the given SUBNET issue The file will be compressed using zstd before upload.

Motivation and Context

For customer's ease of sharing required logs to SUBNET issues

How to test this PR?

mc support upload --issue <issueNum> --comment <msg> <alias> </path/to/file>

Types of changes

Checklist:

klauspost commented 8 months ago

Should we apply compression if not already compressed?

Other than the reduced transfer size, it has the added benefit that it will wrap potentially harmful files and not be directly executable.

The server should not do anything do decompress, but we add a field just to be nice.

Adding it is pretty trivial:

    go func() {
        var (
            part io.Writer
            e    error
        )
        defer func() {
            mwriter.Close()
            w.CloseWithError(e)
        }()

        file, e := os.Open(filePath)
        if e != nil {
            return
        }
        defer file.Close()
        doComp := true
            if strings.HasSuffix(strings.ToLower(filePath), ext) {
                doComp = false
                break
            }
        }
        if doComp {
            mwriter.WriteField("auto-compression", "zstd")
            filePath += ".zst"
        }
        part, e = mwriter.CreateFormFile("file", filepath.Base(filePath))
        if e != nil {
            return
        }

        if doComp {
            z, _ := zstd.NewWriter(part, zstd.WithEncoderConcurrency(2))
            defer z.Close()
            _, e = z.ReadFrom(file)
        } else {
            _, e = io.Copy(part, file)
        }
    }()
anjalshireesh commented 8 months ago

Should we apply compression if not already compressed?

Other than the reduced transfer size, it has the added benefit that it will wrap potentially harmful files and not be directly executable.

The server should not do anything do decompress, but we add a field just to be nice.

Adding it is pretty trivial:

  go func() {
      var (
          part io.Writer
          e    error
      )
      defer func() {
          mwriter.Close()
          w.CloseWithError(e)
      }()

      file, e := os.Open(filePath)
      if e != nil {
          return
      }
      defer file.Close()
      doComp := true
          if strings.HasSuffix(strings.ToLower(filePath), ext) {
              doComp = false
              break
          }
      }
      if doComp {
          mwriter.WriteField("auto-compression", "zstd")
          filePath += ".zst"
      }
      part, e = mwriter.CreateFormFile("file", filepath.Base(filePath))
      if e != nil {
          return
      }

      if doComp {
          z, _ := zstd.NewWriter(part, zstd.WithEncoderConcurrency(2))
          defer z.Close()
          _, e = z.ReadFrom(file)
      } else {
          _, e = io.Copy(part, file)
      }
  }()

Have implemented this. PTAL.