[Enhancement] Gen KCL config from toml config.

[Peefy]

Enhancement

We want to add toml config import for the kcl import tool like JSON and YAML

Reference

• Import YAML to KCL: https://github.com/kcl-lang/kcl-go/blob/main/pkg/tools/gen/genkcl_json.go
• Import JSON to KCL: https://github.com/kcl-lang/kcl-go/blob/main/pkg/tools/gen/genkcl_yaml.go

[XiaoK29]

I tried to use the github.com BurntSushi/toml library referenced in the project to parse and use the Decode function to obtain MetaData to sort the keys, but I found that there were duplicate bugs in parsing

[XiaoK29]

Unordered implementation

package gen

import (
    "bytes"
    "encoding/json"
    "io"
    "strings"

    "github.com/BurntSushi/toml"
    "github.com/goccy/go-yaml"
)

func (k *kclGenerator) genKclFromToml(w io.Writer, filename string, src interface{}) error {
    code, err := readSource(filename, src)
    if err != nil {
        return err
    }

    var tomlMap map[string]interface{}
    if err := toml.Unmarshal(code, tomlMap); err != nil {
        return err
    }

    b, err := json.Marshal(tomlMap)
    if err != nil {
        return err
    }

    yamlData := &yaml.MapSlice{}
    if err = yaml.UnmarshalWithOptions(b, yamlData, yaml.UseOrderedMap(), yaml.UseJSONUnmarshaler()); err != nil {
        return err
    }

    // convert to kcl
    result := convertKclFromYaml(yamlData)

    // generate kcl code
    return k.genKcl(w, kclFile{Config: []config{
        {Data: result},
    }})
}

Orderly implementation, but with bugs

package gen

import (
    "bytes"
    "io"
    "strings"

    "github.com/BurntSushi/toml"
)

func tomlKeysToStringKeys(keys []toml.Key) []string {
    keysStr := make([]string, len(keys))
    for i, key := range keys {
        keysStr[i] = key.String()
    }
    return keysStr
}

func (k *kclGenerator) genKclFromToml(w io.Writer, filename string, src interface{}) error {
    code, err := readSource(filename, src)
    if err != nil {
        return err
    }

    var tomlMap map[string]interface{}
    metaData, err := toml.NewDecoder(bytes.NewReader(code)).Decode(&tomlMap)
    if err != nil {
        return err
    }

    keys := tomlKeysToStringKeys(metaData.Keys())
    return k.genKcl(w, kclFile{Config: []config{
        {Data: convertKclFromToml(tomlMap, "", keys, keys)},
    }})
}

func convertKclFromToml(inputMap map[string]interface{}, prefix string, keys, allKeys []string) []data {
    var result []data

    for _, key := range keys {
        if strings.Contains(key, ".") {
            continue
        }

        currentPrefix := prefix + key + "."

        switch value := inputMap[key].(type) {
        case map[string]interface{}:
            seen := make(map[string]struct{}, len(allKeys))
            subKeys := make([]string, 0, len(value))
            for _, k := range allKeys {
                if strings.HasPrefix(k, currentPrefix) {
                    subKey := strings.TrimPrefix(k, currentPrefix)
                    rootKey := strings.Split(subKey, ".")[0]
                    if _, ok := seen[rootKey]; !ok {
                        subKeys = append(subKeys, rootKey)
                        seen[rootKey] = struct{}{}
                    }
                }
            }
            result = append(result, data{Key: key, Value: convertKclFromToml(value, currentPrefix, subKeys, allKeys)})
        case []map[string]interface{}:
            var vals []interface{}
            for _, v := range value {
                seen := make(map[string]struct{}, len(allKeys))
                subKeys := make([]string, 0, len(value))
                for _, k := range allKeys {
                    if strings.HasPrefix(k, currentPrefix) {
                        subKey := strings.TrimPrefix(k, currentPrefix)
                        rootKey := strings.Split(subKey, ".")[0]
                        if _, ok := seen[rootKey]; !ok {
                            subKeys = append(subKeys, rootKey)
                            seen[rootKey] = struct{}{}
                        }
                    }
                }
                vals = append(vals, convertKclFromToml(v, currentPrefix, subKeys, allKeys))
            }
            result = append(result, data{Key: key, Value: vals})
        default:
            result = append(result, data{Key: key, Value: value})
        }
    }
    return result
}

[Peefy]

Hello @XiaoK29

I think the key in the metadata of the toml library represents the key section in toml rather than the key in the map. When it is a list, it is allowed to appear multiple times, but in reality, it is an alias for a []string.

[XiaoK29]

Hello @XiaoK29

I think the key in the metadata of the toml library represents the key section in toml rather than the key in the map. When it is a list, it is allowed to appear multiple times, but in reality, it is an alias for a []string.

If that's the case, it's not possible to convert it into kcl in an orderly manner

[Peefy]

If that's the case, it's not possible to convert it into kcl in an orderly manner

There are two ways here:

1. Deserialize toml into an ordered structure, reflect the structure, and serialize it into KCL.
2. Directly call the toml parser to traverse all nodes and serialize them as KCL.

[Peefy]

I've given a example for go value -> kcl config with the visitor/walker pattern.

[XiaoK29]

I've given a example for go value -> kcl config with the visitor/walker pattern.我给出了带有访客/步行者模式的 go value -> kcl config 示例。

I don't understand what it means

[XiaoK29]

If that's the case, it's not possible to convert it into kcl in an orderly manner如果是这样的话，就无法有序的转换成kcl了

There are two ways here:这里有两种方法：

1. Deserialize toml into an ordered structure, reflect the structure, and serialize it into KCL.将toml反序列化为有序结构，反映该结构，并序列化为KCL。
2. Directly call the toml parser to traverse all nodes and serialize them as KCL.直接调用toml解析器遍历所有节点并序列化为KCL。

The "github. com/pelletier/go toml" library only provides node parsing function. Currently, the "github. com/BurntSushi/toml" library is in the repository. Do you want me to import "github. com/pelletier/go toml" to do it

[Peefy]

See the toml encoder https://github.com/BurntSushi/toml/blob/master/encode.go and go to kcl encoder:

• https://github.com/kcl-lang/kcl-go/pull/322
• https://github.com/kcl-lang/kcl-go/pull/323

We need to traverse all nodes of toml to generate the corresponding structure of KCL

[XiaoK29]

See the toml encoder https://github.com/BurntSushi/toml/blob/master/encode.go and go to kcl encoder:

• feat: gen kcl config from go value #322
• feat: add more go value marshal to kcl config #323

We need to traverse all nodes of toml to generate the corresponding structure of KCL

Genkcl_value. go cannot guarantee the validity of the key after parsing the input of the toml file. It cannot be sorted based on the original toml file key, but rather alphabetically

[Peefy]

Genkcl_value. go cannot guarantee the validity of the key after parsing the input of the toml file. It cannot be sorted based on the original toml file key, but rather alphabetically

What I mean is that we need to refer to this structure and use the toml parser to implement encoding of the toml syntax tree to KCL configuration, rather than using the results of toml deserialization. If we want to make the results of toml deserialization ordered, we need to use the interface for extension. The native toml library cannot support non sorted results with the toml metadata. https://github.com/BurntSushi/toml/blob/master/decode.go#L21

[Peefy]

I've made a encode function from toml to KCL demo: https://github.com/kcl-lang/kcl-go/pull/324 and you can follow this to impl the decode process.

[Peefy]

Hello @XiaoK29 Are you still working on this issue? If there are really difficulties, perhaps I can take over later in the modification of the toml lib: https://github.com/kcl-lang/kcl-go/blob/main/pkg/3rdparty/toml/decode.go

[XiaoK29]

Hello @XiaoK29 Are you still working on this issue? If there are really difficulties, perhaps I can take over later in the modification of the toml lib: https://github.com/kcl-lang/kcl-go/blob/main/pkg/3rdparty/toml/decode.go

Sorry, I don't have time to deal with it