New (-f -) function issue

[pedroalbanese]

Hi again,

I'm getting an error when I pass the journal via stdin, it says: "Unable to parse date: include".

Thanks in advance.

[howeyc]

the parser only accepts "include" directives if the source is a file. If it's from stdin, shouldn't all transactions come from stdin?

Otherwise just use files.

[pedroalbanese]

But if I include the include parameter in another file, I cannot count the entries in my encrypted journal..

I have part of the transactions encrypted, and another part in CSV. The CSV files are automatically converted to a single file whenever I check the balance (limport/convert), but they remain in CSV (they are shared/editable among my partners). The fact is that I decrypt the encrypted file to stdout*, passing it to the program, but it does not contain the transactions converted to ledger format automatically, so I need the include param calling the file JointCSV.ledger. The include param must be on the encrypted journal, otherwise I cannot obtain the totals.

[howeyc]

I'm really not sure what you are trying to accomplish here. Can't you just temporarily decrypt and put all transactions into one file (to run reports) and the remove the one file once you are done, everything is separate as usual?

So for example, to do what you are doing to decrypt, like "decrypt-stuff file.enc | ledger -f -" becomes decrypt-stuff file.enc > mystuff.plain limport csv-stuff >> mystuff.plain ledger -f mystuff.plain bal

when done, delete mystuff.plain

Also, I'm not sure what you mean by UUID and signature, if you have extra lines in there my tool may not handle it. My file format is much simpler and has fewer features than the C++ version. This is a deliberate choice on my part.

[pedroalbanese]

I understand. My intention is to simplify too. UUID and digital signature was just to illustrate the need to maintain CSV files and not delete them after conversion. I was not referring to the program's functionality. But in this case, you can still add comments with a semicolon.

I just thought that the problem with the include parameter was a bug, not a choice. In this case, it is even possible to pass the journal through stdin, however limitedly (not with all functions), it does not help much. If it is possible to correct just that, I am very grateful.

Yes, it is perfectly possible to decrypt and then calculate, but it is not the safest method and I would have to do it on other people's machines.

Thanks.

[howeyc]

Can you use cat, without having any include directives? Everything should be able to come through stdin this way.

For example, assuming decrypt and then csvledger.dat

decrypt mysecretstuff.enc | cat - csvledger.dat | ledger -f - bal

[pedroalbanese]

This helps a lot! Thank you!

Man, your program is almost perfect, I have just some problems with automation and some temporary solutions to get around some of them. It is a tip if you are going to improve the code. Improvements and tips on how to get around them are very welcome as well.

In order of importance: (the most important are the unsolved ones, of course, very simple ones)

1

-j -J functions (for plotting) very simple. Workaround: (ledger reg | awk "{print $1, $4}" or "$5" repectively, but the accounts and payees cannot contain spaces).

2

-b "last year", "last # months" expressions (for plotting), for dynamic value based on current date. Workaround: [date "+%Y/%m/%d" | awk -F "/" "{$1 = ($1 == 1)? 1 : $1-1} 1" OFS = "/"]

3

limport "note field" parsing to semicolon line. very simple.

4

Scape character to add comments to journal file, suppressed on print report (not semicolon equivalent), as you sayd is a choice, but is necessary

5

"and not \<account>" argument to suppress some subaccount at balance report The workaround is to sum all the subaccounts, except the one you want to omit (which makes things difficult when the number of accounts fluctuates)

6

Pass just total amount to stdout "bal -F% (total)" equivalent Workaround: (awk | tail)

Thank u once again!

[howeyc]

Those are some interesting ideas, but I'm probably not going to do any of them.

Although 3 sounds interesting, I may do that. But likely not.

As for 4, you could strip it yourself? For instance your hidden comment could be ";#hidden comment" cat myfile.ledger | ledger -f - print | sed '/^;#.*/d'

[pedroalbanese]

Nice trick (4).

[pedroalbanese]

Thanks for the improve 3. Perfect!

[howeyc]

This awk should work for number 1:

ledger reg | awk '{ print $1, $NF}' ledger reg | awk '{ print $1, $(NF-1)}'

[pedroalbanese]

Perfect man, thank you so much for the tips.

I have one last question. Exploring the code, I saw which lines were added, so I'm trying to add the "uuid" field, like the note field (enthusiast). I tryed to repeat:

var dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn int
dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn = -1, -1, -1, -1, -1

        // UUID
        if uuidColumn >= 0 && record[uuidColumn] != "" {
            trans.Comments = []string{";" + record[uuidColumn]}
        }

But the output shows just one or another field. Is possible explain how to print 2 commentColumn (2 lines with ;)?

[howeyc]

Perfect man, thank you so much for the tips.

I have one last question. Exploring the code, I saw which lines were added, so I'm trying to add the "uuid" field, like the note field (enthusiast). I tryed to repeat:

var dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn int
dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn = -1, -1, -1, -1, -1

      // UUID
      if uuidColumn >= 0 && record[uuidColumn] != "" {
          trans.Comments = append(trans.Comments, ";" + record[uuidColumn])
      }

But the output shows just one or another field. Is possible explain how to print 2 commentColumn (2 lines with ;)?

adjust as above and make the same adjustment for commentColumn as well

[pedroalbanese]

Where? commentColumn appears only in these 3 places .. can you tell me the line?

var dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn int dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn = -1, -1, -1, -1, -1

    } else if strings.Contains(fieldName, "uuid") {
        uuidColumn = fieldIndex

    // UUID
    if uuidColumn >= 0 && record[uuidColumn] != "" {
        trans.Comments = append(trans.Comments, ";" + record[uuidColumn])
    }

I made the 3 adjustments, but the output always show just 1 line of comment..

[pedroalbanese]

package main

import (
    "encoding/csv"
    "flag"
    "fmt"
    "math/big"
    "os"
    "strings"
    "time"
    "unicode/utf8"

    "github.com/howeyc/ledger"

    "github.com/jbrukh/bayesian"
)

const (
    transactionDateFormat = "2006/01/02"
    displayPrecision      = 2
)

func usage() {
    fmt.Printf("Usage: %s -f <ledger-file> <account> <csv file>\n", os.Args[0])
    flag.PrintDefaults()
    os.Exit(1)
}

func main() {
    var ledgerFileName string
    var accountSubstring, csvFileName, csvDateFormat string
    var negateAmount bool
    var allowMatching bool
    var fieldDelimiter string
    var scaleFactor float64

    flag.BoolVar(&negateAmount, "neg", false, "Negate amount column value.")
    flag.BoolVar(&allowMatching, "allow-matching", false, "Have output include imported transactions that\nmatch existing ledger transactions.")
    flag.Float64Var(&scaleFactor, "scale", 1.0, "Scale factor to multiply against every imported amount.")
    flag.StringVar(&ledgerFileName, "f", "", "Ledger file name (*Required).")
    flag.StringVar(&csvDateFormat, "date-format", "01/02/2006", "Date format.")
    flag.StringVar(&fieldDelimiter, "delimiter", ",", "Field delimiter.")
    flag.Parse()

    ratScale := big.NewRat(1, 1)
    ratScale.SetFloat64(scaleFactor)

    args := flag.Args()
    if len(args) != 2 {
        usage()
    } else {
        accountSubstring = args[0]
        csvFileName = args[1]
    }

    csvFileReader, err := os.Open(csvFileName)
    if err != nil {
        fmt.Println("CSV: ", err)
        return
    }
    defer csvFileReader.Close()

    ledgerFileReader, err := ledger.NewLedgerReader(ledgerFileName)
    if err != nil {
        fmt.Println("Ledger: ", err)
        return
    }

    generalLedger, parseError := ledger.ParseLedger(ledgerFileReader)
    if parseError != nil {
        fmt.Printf("%s:%s\n", ledgerFileName, parseError.Error())
        return
    }

    var matchingAccount string
    matchingAccounts := ledger.GetBalances(generalLedger, []string{accountSubstring})
    if len(matchingAccounts) < 1 {
        fmt.Println("Unable to find matching account.")
        return
    }
    matchingAccount = matchingAccounts[len(matchingAccounts)-1].Name

    allAccounts := ledger.GetBalances(generalLedger, []string{})

    csvReader := csv.NewReader(csvFileReader)
    csvReader.Comma, _ = utf8.DecodeRuneInString(fieldDelimiter)
    csvRecords, cerr := csvReader.ReadAll()
    if cerr != nil {
        fmt.Println("CSV parse error:", cerr.Error())
        return
    }

    classes := make([]bayesian.Class, len(allAccounts))
    for i, bal := range allAccounts {
        classes[i] = bayesian.Class(bal.Name)
    }
    classifier := bayesian.NewClassifier(classes...)
    for _, tran := range generalLedger {
        payeeWords := strings.Split(tran.Payee, " ")
        for _, accChange := range tran.AccountChanges {
            if strings.Contains(accChange.Name, "Expense") {
                classifier.Learn(payeeWords, bayesian.Class(accChange.Name))
            }
        }
    }

    // Find columns from header
    var dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn int
    dateColumn, payeeColumn, amountColumn, commentColumn, uuidColumn = -1, -1, -1, -1, -1
    for fieldIndex, fieldName := range csvRecords[0] {
        fieldName = strings.ToLower(fieldName)
        if strings.Contains(fieldName, "date") {
            dateColumn = fieldIndex
        } else if strings.Contains(fieldName, "description") {
            payeeColumn = fieldIndex
        } else if strings.Contains(fieldName, "payee") {
            payeeColumn = fieldIndex
        } else if strings.Contains(fieldName, "amount") {
            amountColumn = fieldIndex
        } else if strings.Contains(fieldName, "expense") {
            amountColumn = fieldIndex
        } else if strings.Contains(fieldName, "note") {
            commentColumn = fieldIndex
        } else if strings.Contains(fieldName, "uuid") {
            uuidColumn = fieldIndex
        } else if strings.Contains(fieldName, "comment") {
            commentColumn = fieldIndex
        }
    }

    if dateColumn < 0 || payeeColumn < 0 || amountColumn < 0 {
        fmt.Println("Unable to find columns required from header field names.")
        return
    }

    expenseAccount := ledger.Account{Name: "unknown:unknown", Balance: new(big.Rat)}
    csvAccount := ledger.Account{Name: matchingAccount, Balance: new(big.Rat)}
    for _, record := range csvRecords[1:] {
        inputPayeeWords := strings.Split(record[payeeColumn], " ")
        csvDate, _ := time.Parse(csvDateFormat, record[dateColumn])
        if allowMatching || !existingTransaction(generalLedger, csvDate, inputPayeeWords[0]) {
            // Classify into expense account
            _, likely, _ := classifier.LogScores(inputPayeeWords)
            if likely >= 0 {
                expenseAccount.Name = string(classifier.Classes[likely])
            }

            // Negate amount if required
            expenseAccount.Balance.SetString(record[amountColumn])
            if negateAmount {
                expenseAccount.Balance.Neg(expenseAccount.Balance)
            }

            // Apply scale
            expenseAccount.Balance = expenseAccount.Balance.Mul(expenseAccount.Balance, ratScale)

            // Csv amount is the negative of the expense amount
            csvAccount.Balance.Neg(expenseAccount.Balance)

            // Create valid transaction for print in ledger format
            trans := &ledger.Transaction{Date: csvDate, Payee: record[payeeColumn]}
            trans.AccountChanges = []ledger.Account{csvAccount, expenseAccount}

            // Comment
            if commentColumn >= 0 && record[commentColumn] != "" {
                trans.Comments = []string{";" + record[commentColumn]}
            }
            // UUID
            if uuidColumn >= 0 && record[uuidColumn] != "" {
                trans.Comments = []string{";" + record[uuidColumn]}
            }
            PrintTransaction(trans, 79)
        }
    }
}

func existingTransaction(generalLedger []*ledger.Transaction, transDate time.Time, payee string) bool {
    for _, trans := range generalLedger {
        if trans.Date == transDate && strings.HasPrefix(trans.Payee, payee) {
            return true
        }
    }
    return false
}

[howeyc]

your lines

        // Comment
        if commentColumn >= 0 && record[commentColumn] != "" {
            trans.Comments = []string{";" + record[commentColumn]}
        }
        // UUID
        if uuidColumn >= 0 && record[uuidColumn] != "" {
            trans.Comments = []string{";" + record[uuidColumn]}
        }

new lines

        // Comment
        if commentColumn >= 0 && record[commentColumn] != "" {
            trans.Comments = append(trans.Comments, ";" + record[commentColumn])
        }
        // UUID
        if uuidColumn >= 0 && record[uuidColumn] != "" {
            trans.Comments = append(trans.Comments, ";" + record[uuidColumn])
        }

[pedroalbanese]

Very nice man! Now your code meets all my needs. Big thanks! Congratulations on the project, thanks for sharing.

[pedroalbanese]

One issue and a tip (I believe it is simple to do and does not increase the complexity of the code):

7 (Issue)

limport "target account alias" parameter Now that I noticed, the target account is always searched by "Expense" term, this makes it impossible for me to convert CSV containing Incomes .. To deal with this problem I'll have to recompile the code 7 times changing only the term "Expense" to "Income", "Assets", etc. limportass.exe, limportinc.exe, limportinp.exe (inputs), limportmat.exe (material), limportprod.exe (products), etc. 6 executables in addition to the original, what would disturb the portability (I will run it on my Android too); I believe it's no dificult pass this term as optional param substitucting the default search, when necessary. To make the use of limport less restrict.

8 (Tip)

as: -payee string Filter output to payees that contain this string.

an equivalent: -comment string Filter output to "commentlines" that contain this string.

To enable the use of tags, now so easy.

(I thought it better to put it here than open another topic. Sorry to be back so soon. Take your time.)

Thanks!

[pedroalbanese]

Yeah, now it's perfect! Great work man. Big thanks!