search

odeke-em / drive

Google Drive client for the commandline
Apache License 2.0
6.69k stars 427 forks source link

Filenames with extended characters are not handled well #406

Open vaniwaar opened 9 years ago

vaniwaar commented 9 years ago

I have a bunch of music files with extended ascii characters in their names. When I try to push these, drive does not retain the name properly:

vaniwaar@storage ~/gdrive $ drive push -r foo Resolving...

Unfortunately, I can't attach the tar file with this example but I can send it to someone if requested.

--Ron

odeke-em commented 9 years ago

Hello @vaniwaar, thanks for reporting this issue and welcome to drive! So for starters, what does drive version say?

odeke-em commented 9 years ago

Ping!

vaniwaar commented 9 years ago

Sorry for the delayed response.

vaniwaar@storage ~ $ drive version drive version: 0.3.0 Commit Hash: Go Version: OS: BuildTime:

odeke-em commented 9 years ago

Hmm, you can properly bundle the build info by running

$ go get github.com/odeke-em/drive/drive-gen && drive-gen

For sure, please mail the reproducing cases to me and I'll check them out at the end of this week.

vaniwaar commented 9 years ago

Just sent a tar file reproducing this situation.

--Ron

odeke-em commented 9 years ago

Hey @vaniwaar, I got sometime and I've tried to replicate your issue but in vain as you'll see in this YouTube video Failed 406 reproduction

Is there a way you could reliably reproduce this issue?

Thanks.

vaniwaar commented 9 years ago

I will give it a go when I get home. FWIW, the machine on which I am running this is Linux Mint 17.

--Ron

vaniwaar commented 9 years ago

Video of the error occurring for me is at https://drive.google.com/file/d/0B4ia4frohAd6WEUyTnJ3cG53LUU/view?usp=sharing

This is on a Linux box: Linux version 3.13.0-37-generic (buildd@kapok) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014

vaniwaar commented 9 years ago

Didn't mean to close this...

odeke-em commented 9 years ago

The other problem is that you've never run 

$ go get github.com/odeke-em/drive/drive-gen && drive-gen

like I had asked in comment https://github.com/odeke-em/drive/issues/406#issuecomment-147945034. Also you are running code that is more than a month old yet I don't have the commit tag that I had requested for. To get the latest code, please run

$ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gen

then ping me and retry that example. Thanks.

vaniwaar commented 9 years ago

Updated and now have 0.3.1. Tried your command and it crashes:

vaniwaar@storage ~/gdrive/testing $ drive version drive version: 0.3.1 Commit Hash: Go Version: OS: BuildTime: vaniwaar@storage ~/gdrive/testing $ go get -u -v github.com/odeke-em/drive/drive-gen github.com/odeke-em/drive (download) github.com/boltdb/bolt (download) github.com/cheggaaa/pb (download) github.com/mattn/go-isatty (download) github.com/odeke-em/cache (download) github.com/odeke-em/cli-spinner (download) github.com/odeke-em/command (download) github.com/odeke-em/exponential-backoff (download) github.com/odeke-em/extractor (download) github.com/odeke-em/log (download) github.com/odeke-em/meddler (download) github.com/odeke-em/pretty-words (download) github.com/odeke-em/statos (download) github.com/skratchdot/open-golang (download) Fetching https://golang.org/x/net/context?go-get=1 munmap of stack space failed: errno 22 Aborted

goroutine 1 [select]: main.$nested27 /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/http.go:57 main.httpsOrHTTP /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/http.go:65 main.repoRootForImportDynamic /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/vcs.go:468 main.repoRootForImportPath /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/vcs.go:362 main.downloadPackage /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/get.go:269 main.download /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/get.go:166 main.download /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/get.go:232 main.download /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/get.go:232 main.runGet /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/get.go:72 main.main /build/gccgo-go-8UjdDR/gccgo-go-1.2.1/src/cmd/go/main.go:161

goroutine 3 [syscall]: goroutine in C code; stack unavailable

goroutine 8 [finalizer wait]: vaniwaar@storage ~/gdrive/testing $

odeke-em commented 9 years ago

That's a gcc-go issue that you could file against the Go project if you'd like. However, try re-running it as it could be a bunch of factors.

vaniwaar commented 9 years ago

Fixed the crash problem. Now it won't complete due to:

vaniwaar@storage ~ $ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gengithub.com/odeke-em/drive (download) github.com/boltdb/bolt (download) github.com/cheggaaa/pb (download) github.com/mattn/go-isatty (download) github.com/odeke-em/cache (download) github.com/odeke-em/cli-spinner (download) github.com/odeke-em/command (download) github.com/odeke-em/exponential-backoff (download) github.com/odeke-em/extractor (download) github.com/odeke-em/log (download) github.com/odeke-em/meddler (download) github.com/odeke-em/pretty-words (download) github.com/odeke-em/statos (download) github.com/skratchdot/open-golang (download) Fetching https://golang.org/x/net/context?go-get=1 Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200) get "golang.org/x/net/context": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1 get "golang.org/x/net/context": verifying non-authoritative meta tag Fetching https://golang.org/x/net?go-get=1 Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200) golang.org/x/net (download) Fetching https://golang.org/x/oauth2?go-get=1 Parsing meta tags from https://golang.org/x/oauth2?go-get=1 (status code 200) get "golang.org/x/oauth2": found meta tag main.metaImport{Prefix:"golang.org/x/oauth2", VCS:"git", RepoRoot:"https://go.googlesource.com/oauth2"} at https://golang.org/x/oauth2?go-get=1 golang.org/x/oauth2 (download) Fetching https://golang.org/x/oauth2/google?go-get=1 Parsing meta tags from https://golang.org/x/oauth2/google?go-get=1 (status code 200) get "golang.org/x/oauth2/google": found meta tag main.metaImport{Prefix:"golang.org/x/oauth2", VCS:"git", RepoRoot:"https://go.googlesource.com/oauth2"} at https://golang.org/x/oauth2/google?go-get=1 get "golang.org/x/oauth2/google": verifying non-authoritative meta tag Fetching https://golang.org/x/oauth2?go-get=1 Parsing meta tags from https://golang.org/x/oauth2?go-get=1 (status code 200) Fetching https://google.golang.org/cloud/compute/metadata?go-get=1 Parsing meta tags from https://google.golang.org/cloud/compute/metadata?go-get=1 (status code 200) get "google.golang.org/cloud/compute/metadata": found meta tag main.metaImport{Prefix:"google.golang.org/cloud", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://google.golang.org/cloud/compute/metadata?go-get=1 get "google.golang.org/cloud/compute/metadata": verifying non-authoritative meta tag Fetching https://google.golang.org/cloud?go-get=1 Parsing meta tags from https://google.golang.org/cloud?go-get=1 (status code 200) google.golang.org/cloud (download) Fetching https://google.golang.org/api/drive/v2?go-get=1 Parsing meta tags from https://google.golang.org/api/drive/v2?go-get=1 (status code 200) get "google.golang.org/api/drive/v2": found meta tag main.metaImport{Prefix:"google.golang.org/api", VCS:"git", RepoRoot:"https://code.googlesource.com/google-api-go-client"} at https://google.golang.org/api/drive/v2?go-get=1 get "google.golang.org/api/drive/v2": verifying non-authoritative meta tag Fetching https://google.golang.org/api?go-get=1 Parsing meta tags from https://google.golang.org/api?go-get=1 (status code 200) google.golang.org/api (download) Fetching https://google.golang.org/api/googleapi?go-get=1 Parsing meta tags from https://google.golang.org/api/googleapi?go-get=1 (status code 200) get "google.golang.org/api/googleapi": found meta tag main.metaImport{Prefix:"google.golang.org/api", VCS:"git", RepoRoot:"https://code.googlesource.com/google-api-go-client"} at https://google.golang.org/api/googleapi?go-get=1 get "google.golang.org/api/googleapi": verifying non-authoritative meta tag Fetching https://google.golang.org/api?go-get=1 Parsing meta tags from https://google.golang.org/api?go-get=1 (status code 200) github.com/odeke-em/ripper (download) github.com/odeke-em/xon (download) load cmd/cgo: package cmd/cgo: open /usr/src/cmd/cgo: no such file or directory

I c++ all day long but I don't know the first thing about go...

--Ron

odeke-em commented 9 years ago

I once encountered that and after dealing with gcc-go on Ubuntu plus other issues with it, I decided to install gc go from the official Go downloads page https://golang.org/doc/install and not gccgo. Worked for me, feel free to try it.

vaniwaar commented 9 years ago

Getting closer. Now the last few lines are:

github.com/odeke-em/ripper (download) github.com/odeke-em/xon (download) -bash: drive-gengithub.com/odeke-em/drive: No such file or directory vaniwaar@storage ~/go $ vaniwaar@storage ~/go $

If you would like, I could give you an account on my machine and you could just work from there if it is easier...

--Ron

odeke-em commented 9 years ago

No thanks, that's alright. Is your gopath properly set? Since you say it is almost done and the dependencies have been successfully updated

$ go get github.com/odeke-em/drive/drive-gen

Should run if your gopath is properly set. Am off to class for a long day so I might not be too responsive, and no longer at a computer.

vaniwaar commented 9 years ago

Ok, I got things going and the problem is still exactly as it was before:

vaniwaar@storage ~/gdrive/testing $ drive version drive version: 0.3.1 Commit Hash: Go Version: OS: BuildTime: vaniwaar@storage ~/gdrive/testing $ drive push foo Resolving...

odeke-em commented 9 years ago

Hello again @vaniwaar, my mid term exams are finally done and I've gotten a little free time. BTW all along you've never mentioned what your version of Go is.

To test out if this issue is a problem with your setup, golang version, OS packages that aid in string comparison etc, would you mind running this program at https://github.com/odeke-em/go-utils/blob/master/strcmp-check/main.go by

$ go get github.com/odeke-em/go-utils/strcmp-check
$ echo <trippy_filename> | strcmp-check
$ # or
$ cat filenames.txt | strcmp-check

if you see a message like "panic" then it is a problem with your Go version, but please give me information on what the versions are before re-installing etc. Thanks.

vaniwaar commented 9 years ago

Hope your midterms went well! I don't get any output:

vaniwaar@storage ~ $ echo gdrive/testing/foo/98\ Boléro\ (from\ 10).mp3 | strcmp-check vaniwaar@storage ~ $

vaniwaar commented 9 years ago

vaniwaar@storage ~ $ go version go version go1.5.1 linux/amd64 vaniwaar@storage ~ $

odeke-em commented 9 years ago

How about?

$ echo "98 Boléro (from 10).mp3" | strcmp-check
vaniwaar commented 9 years ago

vaniwaar@storage ~ $ echo "98 Boléro (from 10).mp3" | strcmp-check vaniwaar@storage ~ $

odeke-em commented 9 years ago

Edit: Hello @vaniwaar I think @colinkeenan might have diagnosed your problem in issue #472 and he made a wiki entry here https://github.com/odeke-em/drive/wiki

colinkeenan commented 9 years ago

I have updated the wiki (https://github.com/odeke-em/drive/wiki#linux-terminal-character-encoding) with more information about setting LANG. The original as pictured above will work, but I added another way that may be easier:

 sudo localectl set-locale LANG=en_US.UTF-8
odeke-em commented 9 years ago

Thanks again @colinkeenan, in that case I'll remove the screenshot I attached to avoid any confusion so that users can then go directly to the link you've shared.

odeke-em commented 8 years ago

@vaniwaar ping!

vaniwaar commented 8 years ago

Doesn't fix the problem... I have LANG set to en_US.UTF-8 but it still exhibits the same problem.

--Ron

colinkeenan commented 8 years ago

I would like to help determine if this is really a bug or some issue with Linux Mint. I'm not using Linux Mint, but if it works on Arch Linux, Xfce4, Guake terminal, then I don't think it should be considered a bug of Drive.

First though, I'm not clear about what the issue actually is. After pushing the files, do they show up correctly if viewed online in your web browser? Is it just that there are question marks in the text output in the terminal that is the issue?

Can you send me an empty file with a problematic name so I can test it on Arch Linux, Xfce desktop, in the Guake terminal. If the issue is just with terminal output, I can also install whatever terminal you are using and test it in that.

odeke-em commented 8 years ago

Thank you for stepping up to help solve this @colinkeenan! So to reproduce it, @vaniwaar once emailed me this tar at https://drive.google.com/open?id=0By5qQkvRAeV2MzFqTm50NVV3NW8

$ shasum -a 256 error.tar # Verify the intergrity of the tar
25774062935de135d6062fb6afe649c0a525437a4e268e6d9c64b6e7454e4f3e  error.tar

and in the issue's preceeding comments I made a short clip of me failing to reproduce his issue.

colinkeenan commented 8 years ago

I am having the same problem as @vaniwaar. The problem seems to be that the file is not actually encoded as UTF-8 though. When I look at it in the Xfce file manager Thunar, it shows up as 98 Bol�ro (from 10).mp3 (invalid encoding). I have been researching how the title is encoded, but I'm not getting anywhere.

The "bug" may not be what it seems though. I just tried to use the web interface to upload the file, and I get "An internal server error occured". I tried several times with the same result. I tried other files, and they uploaded just fine.

It seems that what drive should actually do in this case is recognize that the uploaded file name doesn't really match the local file name and report the error, refusing to upload.

colinkeenan commented 8 years ago

I have found a way to upload the test file properly. I had to change the encoding to UTF-8 though:

convmv -f iso-8859-14 -t utf8 --notest 98*

where, of course, 98* expands to the full test filename.

If you run convmv without --notest, it will do a "dry run" without actually changing anything.

Edit: I used iso-8859-14 just because that's what Mousepad suggested. Any of the iso-8859 series would have worked. The most common are iso-8859-1 and iso-8859-15. See https://en.wikipedia.org/wiki/ISO/IEC_8859#The_Parts_of_ISO.2FIEC_8859

colinkeenan commented 8 years ago

The following is interesting although probably not helpful: before converting the filename to utf-8, I tried reading the filename in various editors. First, I tried editing the directory itself vim Downdoads and it showed the question mark. Then, I tried echo Downloads/98* > enc.txt and vim enc.txt. To my surprise, it looked fine: 98 Boléro (from 10).mp3. But cat enc.txt still had the question mark. I then tried the Xfce editor Mousepad, and it detected the problem with the encoding, offering iso-8859-14 as an alternative which worked. I also tried Notepad running in Wine and it worked without asking questions, just like Vim did.

colinkeenan commented 8 years ago

As an alternative to converting the files to utf-8, you could change your environment. I found the following works:

  1. edit (with sudo ...) /etc/locale.gen and uncomment en_IE@euro ISO-8859-15 (if already uncommented, skip next step)
  2. sudo locale-gen
  3. sudo localectl set-locale LANG=en_IE.ISO-8859-15@euro
  4. reboot

Now the files will show correctly with ls and should upload correctly with drive push although I didn't test it.

vaniwaar commented 8 years ago

If /etc/locale.gen does not exist, what is the alternative?

--ROn

colinkeenan commented 8 years ago

First, maybe an ISO-8859 character encoding environment is already available to you. Check what's available with:

locale -a

Then, you can skip to the localectl step or edit the relevant file directly.

colinkeenan commented 8 years ago

Ok, I've found a Linux Mint Forum post explaining how to do this kind of thing in Linux Mint. It's not easy (at least as of early 2013). I don't think Linux Mint has switched to systemd yet. If you don't already have any ISO-8859 stuff showing up with locale-a, here is how you can generate the locale by hand according to an answer in this post: http://forums.linuxmint.com/viewtopic.php?f=47&t=123673. He's using gedit, so I'm just copying his answer, but you could of course use whatever editor you want and if a terminal editor, just sudo instead of gksudo.

  1. gksudo gedit /var/lib/locales/supported.d/local
  2. Add this to it: en_US.ISO-8859-1 ISO-8859-1, save and exit.
  3. sudo dpkg-reconfigure locales
  4. gksudo gedit /etc/default/locale
  5. Put this in it: LANG=en_US.ISO-8859-1

If you only want to use LANG=en_US.ISO-8859-1 when running the terminal, you can skip steps 4 and 5. Then, run the terminal with LANG=en_US.ISO-8859-1 x-terminal-emulator. Of course, you can use whatever terminal you like rather than x-terminal-emulator. That's just what I could find for Linux Mint.

odeke-em commented 8 years ago

@colinkeenan thanks for looking into this fist bump! Happy thanksgiving too.

odeke-em commented 8 years ago

@vaniwaar ping!

vaniwaar commented 8 years ago

Still see the problem. I made the modifications as suggested by collinkeenan above but see exactly the same behavior. Here is the relevant changes:

vaniwaar@storage ~ $ cat /var/lib/locales/supported.d/local en_US.UTF-8 UTF-8 en_US.ISO-8859-1 ISO-8859-1 vaniwaar@storage ~ $ cat /etc/default/locale LANG=en_US.ISO-8859-1 vaniwaar@storage ~ $ locale -a C C.UTF-8 en_AG en_AG.utf8 en_AU.utf8 en_BW.utf8 en_CA.utf8 en_DK.utf8 en_GB.utf8 en_HK.utf8 en_IE.utf8 en_IN en_IN.utf8 en_NG en_NG.utf8 en_NZ.utf8 en_PH.utf8 en_SG.utf8 en_US.iso88591 en_US.utf8 en_ZA.utf8 en_ZM en_ZM.utf8 en_ZW.utf8 POSIX

odeke-em commented 8 years ago

@vaniwaar I think this issue is beyond drive's scope and is with the user's locale's as @colinkeenan has diagnosed.

odeke-em commented 8 years ago

Ping!

odeke-em commented 8 years ago

I'll be closing this issue shortly as it isn't related to drive and a diagnosis and remedy was provided already.

vaniwaar commented 8 years ago

Go ahead and close it but I disagree since the problem is detectable and should at least throw an error. I have been able to remedy it here and have finally just renamed the files.

--Ron

colinkeenan commented 8 years ago

@vaniwaar did you see my instructions for converting each file to utf-8 so that even though "renamed" they look exactly the same with extended characters etc?

convmv -f iso-8859-15 -t utf8 --notest 98*

@odeke-em I agree with @vaniwaar that it should detect the problem (that the server name does not actually match the local name) and provide an error message.

vaniwaar commented 8 years ago

@colinkeenan, I missed that instruction. Thanks!

--Ron

odeke-em commented 8 years ago

@vaniwaar good point. @colinkeenan good point, we could add a sanity check at the beginning.

odeke-em commented 8 years ago

Hey folks, I was wondering if y'all could try out the output of this https://gist.github.com/odeke-em/3f8eabb5547168fd402b and see if it catches your invalid locale. If so, we can adapt this code into drive to help catch this cases.

colinkeenan commented 8 years ago

It was in an inconvenient format because I assume I was not to include the "$" and ">" at the beginning of each line. I copy and pasted it line by line leaving that first character off. I got an error though:

/home/colin %mkdir -p locale-test
 /home/colin %cd locale-test && cat << ! > main.go && go get . && go run .
cmdand cmdand cmdand heredoc> package main
cmdand cmdand cmdand heredoc> 
cmdand cmdand cmdand heredoc> // Code credited to and adapted from asciinema
cmdand cmdand cmdand heredoc> // https://github.com/asciinema/asciinema/blob/c3ba15ba56aae69dd91d9a82a58d5351ad6f0285/main.go
cmdand cmdand cmdand heredoc> 
cmdand cmdand cmdand heredoc> import (
cmdand cmdand cmdand heredoc>   "fmt"
cmdand cmdand cmdand heredoc>   "os"
cmdand cmdand cmdand heredoc>   "strings"
cmdand cmdand cmdand heredoc> 
cmdand cmdand cmdand heredoc>   "github.com/asciinema/asciinema/util"
cmdand cmdand cmdand heredoc> )
cmdand cmdand cmdand heredoc> 
cmdand cmdand cmdand heredoc> func environment() map[string]string {
cmdand cmdand cmdand heredoc>   env := map[string]string{}
cmdand cmdand cmdand heredoc>   for _, keyval := range os.Environ() {
cmdand cmdand cmdand heredoc>           pair := strings.SplitN(keyval, "=", 2)
cmdand cmdand cmdand heredoc>           env[pair[0]] = pair[1]
cmdand cmdand cmdand heredoc>   }
cmdand cmdand cmdand heredoc>   return env
cmdand cmdand cmdand heredoc> }
cmdand cmdand cmdand heredoc> 
cmdand cmdand cmdand heredoc> func main() {
cmdand cmdand cmdand heredoc>   env := environment()
cmdand cmdand cmdand heredoc>   if !util.IsUtf8Locale(env) {
cmdand cmdand cmdand heredoc>           fmt.Fprintf(os.Stderr, "a non UTF-8 native locale detected")
cmdand cmdand cmdand heredoc>           os.Exit(1)
cmdand cmdand cmdand heredoc>   }
cmdand cmdand cmdand heredoc>  }
cmdand cmdand cmdand heredoc> !
go install: no install location for directory /home/colin/locale-test outside GOPATH
    For more details see: go help gopath
 /home/colin/locale-test %
odeke-em commented 8 years ago

Ooops my bad, let me it in the gist, please see https://gist.github.com/odeke-em/3f8eabb5547168fd402b#file-main-go. Also if you use Vi/Vim you could just say

:%s/^> / /g

and it will strip them out

odeke-em commented 8 years ago

Yeah please properly setup your $GOPATH as shown here https://github.com/odeke-em/drive#requirements