[Research] File Manager

[Akecel]

Faire des recherche sur les files manager des framework GO existant : Echo :

• https://echo.labstack.com/cookbook/file-upload/
• https://echo.labstack.com/cookbook/file-download/ Gin :
• https://gin-gonic.com/docs/examples/upload-file/

L'idée est de voir comment ses framework mette cela en place, es-ce qu'ils utilisent juste le module HTTP Go et le surcharge avec des helper plus lisible ou font il des choses un peu plus poussé Es-ce que c'est nécessaire de développer ce genre de module au sein de notre framework ou peut ont se contenter de ce qui est dispo nativement ?

Drop des docs, des liens en commentaire et un petit texte sur ce que tu en pense et on verra ça tous ensemble avec tes conclusion pour décider ce qu'on fait (abandon, developpement, ou mise en pause selon le temps)

👋

[JackMaarek]

Plan

This document aims to define the technical research in order to have an overview of the different process implemented by our competitors (Gin, Echo).

Goal

As we are creating a framework, the ability of uploading or downloading files is necessary. This task is dedicated on analizing the technical implementations in the different http frameworks written in Go.

Concerned ressource

• HTTP Upload/Download

Workflow

• Use a file upload or download route.
• Upload / Download a file.
• Let the file manager handle the rest.

Overview

Upload

Gin

https://github.com/gin-gonic/gin/blob/master/context.go::577 (FormFile)

File is gathered by the request from http package, it parses the multipart form from the associated key.

https://github.com/gin-gonic/gin/blob/master/context.go::577 (SaveUploadedFile)

This method only uses base function of go, it is just a wrapper function implemented by the Context struct. Should be ok to develop it on our end in order to modify this method allowing us to upload file directly to CDN.

This method is taking as parameter the pointer file *multipart.FileHeader gathered by FormFile function and the destination string path It uses the file to read it, creates the destination directory/file and copy it in the file system.

Echo

https://github1s.com/labstack/echo/blob/HEAD/context.go ::383 (FormFile)

Same as above except of the wrapping logic.

Here the process is different, the file is given by the http request itself it uses the Context wrapper struct with the Request *http.Request propery to have access directly from Context. Once accessed the logic is pretty much the same as Gin's.

Download

Gin

Gin does not even implement this method.

Echo

https://github1s.com/labstack/echo/blob/HEAD/context.go ::571 (File)

This method also uses basic methods from the http package. Still ok to redevelop on out end.

It only opens the file, proceed to basic checks, and uses http.ServeContent method in order to return the opened file.

Recomendations

• Redeveloping this whole service is essential in order to have complete control of the workflow.
• We should use a CDN to store our files in order to avoid poluting the file system.
• To prevent us from vulnerabilities of malicious threats some security guidelines should be followed. (Content-Type Validation, File Signature Validation, Filename Sanitization, and more.)

Sources

• native Upload file tutorial

  https://tutorialedge.net/golang/go-file-upload-tutorial/

• OWASP Recos on file uploading

  https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html

[JackMaarek]

package main

import (
    "io"
    "mime/multipart"
    "net/http"
    "os"
)

var MaxMemory int64 = 32 << 20 // 32MB

type SomeBaseStruct struct {
    Request http.Request
}

// HandleFile allow you to parse the request's multipart form by the file key and return the file.
func (sbs *SomeBaseStruct) HandleFile(formKey string) (*multipart.FileHeader, error)  {
    if sbs.Request.MultipartForm == nil {
        if err := sbs.Request.ParseMultipartForm(MaxMemory); err != nil {
            return nil, err
        }
    }
    f, fh, err := sbs.Request.FormFile(formKey)
    if err != nil {
        return nil, err
    }

    f.Close()
    return fh, nil
}

// Upload allow you to gather the file from the submitted form and copy the passed file to the system directory.
func (sbs *SomeBaseStruct) Upload(formKey string, dst string) error {
    fh, err := sbs.HandleFile(formKey)
    if err != nil {
        return err
    }

    src, err := fh.Open()
    if err != nil {
        return err
    }
    defer src.Close()

    out, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer out.Close()

    _, err = io.Copy(out, src)
    return err

}

// Download allow you to fetch a file from a designated path.
func (sbs SomeBaseStruct) Download(filePath string) (*os.File, error) {
    f, err := os.Open(filePath)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    fi, _ := f.Stat()
    if fi.IsDir() {
        f, err = os.Open(filePath)
        if err != nil {
            return nil, err
        }
        defer f.Close()
        if fi, err = f.Stat(); err != nil {
            return nil, err
        }
    }
    return f, nil
}