search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.27k stars 17.7k forks source link

archive/zip: add support for extended timestamps #18359

Closed bobjalex closed 7 years ago

bobjalex commented 7 years ago

What version of Go are you using (go version)?

go version go1.8beta2 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64 set GOBIN= set GOEXE=.exe set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOOS=windows set GOPATH=C:\GoLib_beta set GORACE= set GOROOT=C:\Go_beta set GOTOOLDIR=C:\Go_beta\pkg\tool\windows_amd64 set GCCGO=gccgo set CC=gcc set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Bob\AppData\Local\Temp\go-build778111039=/tmp/go-build -gno-record-gcc-switches set CXX=g++ set CGO_ENABLED=1 set PKG_CONFIG=pkg-config set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2

What did you do?

I use zip archives in contexts that require correct file times, and are required to be compatible with non-Go zip libraries. Go's default archive/zip writing is incompatible with many other zip libraries and programs (e.g. Java's and Python's provided libraries, the info-zip zip program distributed with Linux, ...).

The problem:

Some solutions: (1) Change archive/zip to store local non-extended times.

I favor (2), since it is flexible and would not violate the Go 1 principle. Default operation would be exactly as pre-Go 8 versions.

What did you expect to see?

Extended time stored as UTC and old (main) time fields stored as local time.

What did you see instead?

All file times stored as UTC.

dsnet commented 7 years ago

This was something that was annoying me about the current zip.FileHeader. I think it was an API mistake for there to be a ModifiedTime and ModifiedDatefield storing the time as MS-DOS formatted uint16s.

The path forward is probably to introduce another field ModTime time.Time and deprecate the MS-DOS timestamps in the same way we dealt with the uint32 sized fields.

EDIT: We'll need a different name since ModTime is already taken as the field name.

dsnet commented 7 years ago

Also, to understand your problem more clearly. Is this problem something blocking you for Go 1.8 release? I don't think we can improve time handling in the zip package for the remainder of Go 1.8, but if this is problematic, we can revert the current change and go for a more complete round-trip fix in Go 1.9.

bradfitz commented 7 years ago

Let's partially roll this back for 1.8 and fix it fully in 1.9.

bobjalex commented 7 years ago

This problem is not blocking me -- I have several ways to work around it.

In fiddling with this problem, I created a new zip module derived from the 1.8 one that solves the problem and is compatible with older versions. I added 2 new function that allow specification of whether the MSDOS 16-bit time and date fields should store as UTC (as it does now) or local (as many existing zip programs and libraries do): func FileInfoHeaderLocal(fi os.FileInfo, localTime bool) (FileHeader, error) // for writing func OpenReaderLocal(name string, localTime bool) (ReadCloser, error) // for reading

That seems to be all that is necessary (with an handful of other hidden code changes to make that work, but not very much, really).

My new version seems to be working OK for my purposes, but has not been exhaustively QA'd.

I attempted to attach my code -- feel free to use these changes if suitable, or point out improvements.

newzip.zip

dsnet commented 7 years ago

In light of #18378, I'm planning on reverting the whole change and just fix it fully for Go 1.9.

gopherbot commented 7 years ago

CL https://golang.org/cl/34651 mentions this issue.

dsnet commented 7 years ago

Re-purposing this issue to be about adding full support for extended timestamps. The solution we come up with should:

bobjalex commented 7 years ago

Reposting the modified archive/zip code I posted in this thread with changes to handle local time in the 16-bit timestamp fields, to allow compatibility with much existing zip software. Found and fixed a bug in setting proper file time in some unzipped files. Now it's perfect :-) newzip.zip

dsnet commented 7 years ago

I'll take a look at this when the Go1.9 development starts up. Thanks for your help.

mattn commented 7 years ago

@dsnet I hope this issue fix. And be sure in come in go1.9 .

Probably add a new field of type time.Time to FileHeader

What the reason for this? As my understanding, Go still have problem that doesn't handle extended file timestamp. And it should be fixed. So I think we don't have to keep compatibility.

bobjalex commented 7 years ago

As work on this commences, I'd like to restate what I've said before in this thread about the main, non-extended timestamp.

The current archive/zip implementation unconditionally stores UTC time in the zip-original non-extended ms-dos-like timestamp field. This causes incompatibility with many (probably most) existing zip programs and libraries as they expect local time in that field. The zip "standard" does not specify whether local or UTC should be stored, but is seems the ad-hoc standard is to use local. Many implementations ignore the optional timestamp fields, so it is important that the only non-optional timestamp contains what is usually expected.

Please, either change to store local time in the non-extended timestamp, or provide a mechanism so that it's at least possible to do so.

The various enhanced zip timestamp optional fields are always UTC (I think), so no issue there.

Thanks, Bob

On Sun, Feb 5, 2017 at 6:23 PM, mattn notifications@github.com wrote:

@dsnet https://github.com/dsnet I hope this issue fix. And be sure in come in go1.9 .

Probably add a new field of type time.Time to FileHeader

What the reason for this? As my understanding, Go still have problem that doesn't handle extended file timestamp. And it should be fixed. So I think we don't have to keep compatibility.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-277572673, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAVdo6wsKRhtnVt_89Dj-XLU6QFY8ks5rZoQlgaJpZM4LPvG1 .

mattn commented 7 years ago

Please, either change to store local time in the non-extended timestamp, or provide a mechanism so that it's at least possible to do so.

Store local-time in the non-extended timestamp is not good idea. It will make problem. When make zip file at 00:00 local time in several country, the timestamp should always pointed different time. Because it should have time offset. The field always be stored UTC.

In my opinion, the timestamp handled extra-field MUST be default in archive/zip. For example, unzip command show the offset-ed timestamp always with unzip -l.

c:\dev\unzip\unzip60\tmp>dir ..\README
 Volume in drive C has no label.
 Volume Serial Number is 207E-613D

 Directory of c:\dev\unzip\unzip60

2009/04/20  07:37            18,337 README
               1 File(s)         18,337 bytes
               0 Dir(s)  262,483,783,680 bytes free

c:\dev\unzip\unzip60\tmp>zip test.zip ..\README
  adding: ../README (deflated 58%)

c:\dev\unzip\unzip60\tmp>unzip -l test.zip
Archive:  test.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    18337  2009-04-20 07:37   ../README
---------                     -------
    18337                     1 file

c:\dev\unzip\unzip60\tmp>unzip test.zip
Archive:  test.zip
warning:  skipped "../" path component(s) in ../README
  inflating: README

c:\dev\unzip\unzip60\tmp>dir README
 Volume in drive C has no label.
 Volume Serial Number is 207E-613D

 Directory of c:\dev\unzip\unzip60\tmp

2009/04/20  07:37            18,337 README
               1 File(s)         18,337 bytes
               0 Dir(s)  262,483,570,688 bytes free

BTW, What is the usecase that you want to handle non-extended timestamp?

If there are people who want to use non-extended timestamp, we may have to provide API to access non-extended timestamp. But I think extended timestamp should be default (f.ModTime). The code always behave as same in anywhere, anytimes. We shouldn't add workaround adding time offset in code of application.

bobjalex commented 7 years ago

Hi mattn --

I didn't suggest storing local time in the extended time field. I agree, that would be wrong.

My suggestion is to store local time in the "main" time field only. Other zip extra time fields should be stored as they are specified, which is always UTC (I think).

That is different from the way archive/zip works now -- now it always stores UTC in the main time field. This causes it to be file-time incompatible with most other zip implementations.

Looking at Go's future, with support added for the "extended time" extra field, I believe that archive/zip should store local time in the main time field and UTC in the extended time field (as specified for the extended time field).

That would allow Go to be more inter-operable with the non-Go world, but would cause 1.9-created zip files to be file-time incompatible with 1.8 and earlier Go versions. To allow 1.9 programs to be compatible with older Go-created zip files, the revised package should probably provide an option to work with zip files using either local time or UTC in the main time field.

I'm assuming that a goal is that the Go zip package should be compatible with as many existing zip producers and consumers as possible, not just with other Go programs. (If the latter is the case, my suggestion is irrelevant.)

The use case for using the non-extended timestamp is that the extended timestamp is optional and many zip implementations don't use it -- either because they are older implementations that don't know about extended timestamps, or they just want to keep their implementations simple and don't need the greater capability of extended timestamps. For example, the zip implementations in Java's java.util.zip package and Python's zipfile module do not use extended timestamps. Common zip programs such as Info-ZIP, and PKWare do support extended timestamps, but store local times in the main timestamp field when zipping, and expect local times when unzipping. If Go doesn't also do that, it does not play nicely with those programs.Go programs should be able to produce zip files without knowing which external implementation might read it, and conversely, be able to consume zip files produced by a variety of external implementations.

So, my suggestion summarized:

Some notes:

With the current archive/zip it is possible to effect local times by offsetting the time appropriately in the calling program on zip file creation, and reversing the offset on unzipping. I am currently doing that successfully.

With the new archive/zip sent out with 1.8beta1, the above trick no longer worked. Offsetting the time to get local also caused local to be stored in the extended timestamp. On unzipping by non-Go implementations, the retrieved file time would be different depending on whether the non-Go implementation supports extended time stamps or not. As previously noted, not all implementations in the wild support extended time stamps, such as Java programs using java.util.zip and Python programs using zipfile. There was no way to create zip files that a variety of unzipping programs could decode with proper file times. (So I created my own zip package based on 1.8beta1 archive/zip that had the behavior I want. But alas, that is now unnecessary since the 1.8beta1 archive/zip was discarded.)

The previous paragraph shows that it is possible for a zip package to be incapable of writing zip files that play nicely with the variety of zip implementations that are out there. I encourage that the next archive/zip be sufficiently flexible to maximize compatibility.

Sorry this reply got so long...

On Mon, Feb 6, 2017 at 5:14 PM, mattn notifications@github.com wrote:

Please, either change to store local time in the non-extended timestamp, or provide a mechanism so that it's at least possible to do so.

Store local-time in the non-extended timestamp is not good idea. It will make problem. When make zip file at 00:00 local time in several country, the timestamp should always pointed different time. Because it should have time offset. The field always be stored UTC.

In my opinion, the timestamp handled extra-field MUST be default in archive/zip. For example, unzip command show the offset-ed timestamp always with unzip -l.

c:\dev\unzip\unzip60\tmp>dir ..\README Volume in drive C has no label. Volume Serial Number is 207E-613D

Directory of c:\dev\unzip\unzip60

2009/04/20 07:37 18,337 README 1 File(s) 18,337 bytes 0 Dir(s) 262,483,783,680 bytes free

c:\dev\unzip\unzip60\tmp>zip test.zip ..\README adding: ../README (deflated 58%)

c:\dev\unzip\unzip60\tmp>unzip -l test.zip Archive: test.zip Length Date Time Name

18337  2009-04-20 07:37   ../README
18337                     1 file

c:\dev\unzip\unzip60\tmp>unzip test.zip Archive: test.zip warning: skipped "../" path component(s) in ../README inflating: README

c:\dev\unzip\unzip60\tmp>dir README Volume in drive C has no label. Volume Serial Number is 207E-613D

Directory of c:\dev\unzip\unzip60\tmp

2009/04/20 07:37 18,337 README 1 File(s) 18,337 bytes 0 Dir(s) 262,483,570,688 bytes free

BTW, What is the usecase that you want to handle non-extended timestamp?

If there are people who want to use non-extended timestamp, we may have to provide API to access non-extended timestamp. But I think extended timestamp should be default (f.ModTime). The code always behave as same in anywhere, anytimes. We shouldn't add workaround adding time offset in code of application.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-277869500, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNASnKQDDE2k0yIo06fpY3l-O02TBIks5rZ8V0gaJpZM4LPvG1 .

bobjalex commented 7 years ago

Hey mattn -- I just realized that I misread your reply, and my previous message is a bit off-base because of that. Still, please consider my suggestion to at least optionally allow local time in the main file time field.

I agree that putting UTC in the main time field is a better choice, but unfortunately the zipfile world has historically made its ad hoc standard to use local time in that field. Way back when the zip format started, non-local time zones were not as important as they are now.

It's a question of better compatibility versus doing the what is "more logical".

My personal use case is to be able to create and consume zip files inter-operable with Java programs, Python programs, Info-ZIP utilities, and as many other zip implementations as possible, yielding correct file dates.

On Tue, Feb 7, 2017 at 10:28 AM, Bob Alexander bobjalex@gmail.com wrote:

Hi mattn --

I didn't suggest storing local time in the extended time field. I agree, that would be wrong.

My suggestion is to store local time in the "main" time field only. Other zip extra time fields should be stored as they are specified, which is always UTC (I think).

That is different from the way archive/zip works now -- now it always stores UTC in the main time field. This causes it to be file-time incompatible with most other zip implementations.

Looking at Go's future, with support added for the "extended time" extra field, I believe that archive/zip should store local time in the main time field and UTC in the extended time field (as specified for the extended time field).

That would allow Go to be more inter-operable with the non-Go world, but would cause 1.9-created zip files to be file-time incompatible with 1.8 and earlier Go versions. To allow 1.9 programs to be compatible with older Go-created zip files, the revised package should probably provide an option to work with zip files using either local time or UTC in the main time field.

I'm assuming that a goal is that the Go zip package should be compatible with as many existing zip producers and consumers as possible, not just with other Go programs. (If the latter is the case, my suggestion is irrelevant.)

The use case for using the non-extended timestamp is that the extended timestamp is optional and many zip implementations don't use it -- either because they are older implementations that don't know about extended timestamps, or they just want to keep their implementations simple and don't need the greater capability of extended timestamps. For example, the zip implementations in Java's java.util.zip package and Python's zipfile module do not use extended timestamps. Common zip programs such as Info-ZIP, and PKWare do support extended timestamps, but store local times in the main timestamp field when zipping, and expect local times when unzipping. If Go doesn't also do that, it does not play nicely with those programs.Go programs should be able to produce zip files without knowing which external implementation might read it, and conversely, be able to consume zip files produced by a variety of external implementations.

So, my suggestion summarized:

  • allow either local time or UTC be be stored in the main time field, according to option expressed by the caller
  • always store UTC in the extended time field (as specified for that extra field), independent of what is stored in the main time field

Some notes:

With the current archive/zip it is possible to effect local times by offsetting the time appropriately in the calling program on zip file creation, and reversing the offset on unzipping. I am currently doing that successfully.

With the new archive/zip sent out with 1.8beta1, the above trick no longer worked. Offsetting the time to get local also caused local to be stored in the extended timestamp. On unzipping by non-Go implementations, the retrieved file time would be different depending on whether the non-Go implementation supports extended time stamps or not. As previously noted, not all implementations in the wild support extended time stamps, such as Java programs using java.util.zip and Python programs using zipfile. There was no way to create zip files that a variety of unzipping programs could decode with proper file times. (So I created my own zip package based on 1.8beta1 archive/zip that had the behavior I want. But alas, that is now unnecessary since the 1.8beta1 archive/zip was discarded.)

The previous paragraph shows that it is possible for a zip package to be incapable of writing zip files that play nicely with the variety of zip implementations that are out there. I encourage that the next archive/zip be sufficiently flexible to maximize compatibility.

Sorry this reply got so long...

On Mon, Feb 6, 2017 at 5:14 PM, mattn notifications@github.com wrote:

Please, either change to store local time in the non-extended timestamp, or provide a mechanism so that it's at least possible to do so.

Store local-time in the non-extended timestamp is not good idea. It will make problem. When make zip file at 00:00 local time in several country, the timestamp should always pointed different time. Because it should have time offset. The field always be stored UTC.

In my opinion, the timestamp handled extra-field MUST be default in archive/zip. For example, unzip command show the offset-ed timestamp always with unzip -l.

c:\dev\unzip\unzip60\tmp>dir ..\README Volume in drive C has no label. Volume Serial Number is 207E-613D

Directory of c:\dev\unzip\unzip60

2009/04/20 07:37 18,337 README 1 File(s) 18,337 bytes 0 Dir(s) 262,483,783,680 bytes free

c:\dev\unzip\unzip60\tmp>zip test.zip ..\README adding: ../README (deflated 58%)

c:\dev\unzip\unzip60\tmp>unzip -l test.zip Archive: test.zip Length Date Time Name

18337  2009-04-20 07:37   ../README
18337                     1 file

c:\dev\unzip\unzip60\tmp>unzip test.zip Archive: test.zip warning: skipped "../" path component(s) in ../README inflating: README

c:\dev\unzip\unzip60\tmp>dir README Volume in drive C has no label. Volume Serial Number is 207E-613D

Directory of c:\dev\unzip\unzip60\tmp

2009/04/20 07:37 18,337 README 1 File(s) 18,337 bytes 0 Dir(s) 262,483,570,688 bytes free

BTW, What is the usecase that you want to handle non-extended timestamp?

If there are people who want to use non-extended timestamp, we may have to provide API to access non-extended timestamp. But I think extended timestamp should be default (f.ModTime). The code always behave as same in anywhere, anytimes. We shouldn't add workaround adding time offset in code of application.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-277869500, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNASnKQDDE2k0yIo06fpY3l-O02TBIks5rZ8V0gaJpZM4LPvG1 .

mattn commented 7 years ago

Well, I want to fix this until 1.9 . If someone will go to right way instead me, please do it. Or please tell me what is right fix. I'm probably confusing.

bobjalex commented 7 years ago

Hi mattn --

My request is simply that the user of the archive/zip package has the option of storing local time in the "main" (non-extra) time field. The more modern "extra" time fields are specified to always be UTC, so no issue with those.

As you probably know, the existing archive/zip package always stores and expects UTC in the main time field, and does not support any of the 3 or 4 "extra" time fields defined in the current zip ad hoc standard ( https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.3.TXT). That's OK for just storing and reading your own archives, but can be a problem when interoperating with other existing implementations, most (I believe) of which expect local time in the main time field

The local time issue can be worked around with the current archive/zip version: it is possible (but messy) to interface with legacy zip usage by offsetting the times given to and received from the zip package appropriately to make the stored time local.

The problem occurs when the package supports one or more of the "extra" time fields (as the early 1.8 version did). When extra fields are also supported, offsetting the time does not work -- it still makes the main time field local, but causes the extra field(s) to be wrong. With that 1.8 implementation there was no way to have the main field be local and the extra field be UTC.

When trying to be compatible with zip files created by a variety of existing zip implementations it is important that all provided time fields be consistent. As the zip "standard" evolved, implementations tended to prefer one of the extra fields, falling back to the main field if the extra field is not present. So some implementations will see an extra time field and some will see the main field for the same zip file. When using the main time field, implementations usually expect local time and make adjustments as necessary.

It seems to me that the best way to please everyone is to provide the capability of local or UTC in the main time field.

Bob

On Fri, May 26, 2017 at 3:57 AM, mattn notifications@github.com wrote:

Well, I want to fix this until 1.9 . If someone will go to right way instead me, please do it. Or please tell me what is right fix. I'm probably confusing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-304253662, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAdfeYYOkF_dNPpcUjTkmaUhJAeTUks5r9rAggaJpZM4LPvG1 .

mattn commented 7 years ago

As far as I can read your opinion, I guess that you mean below.

When zip timestamp is written as "2016/01/02 15:04:05", ft.ModTime() should be 2016/01/02 15:04:05 JST for japanese local.

Right? Then patch will be below.

diff --git a/src/archive/zip/reader_test.go b/src/archive/zip/reader_test.go
index dfaae78436..8a09f897bd 100644
--- a/src/archive/zip/reader_test.go
+++ b/src/archive/zip/reader_test.go
@@ -15,7 +15,6 @@ import (
    "regexp"
    "strings"
    "testing"
-   "time"
 )

 type ZipTest struct {
@@ -369,13 +368,8 @@ func readTestFile(t *testing.T, zt ZipTest, ft ZipTestFile, f *File) {
    }

    if ft.Mtime != "" {
-       mtime, err := time.Parse("01-02-06 15:04:05", ft.Mtime)
-       if err != nil {
-           t.Error(err)
-           return
-       }
-       if ft := f.ModTime(); !ft.Equal(mtime) {
-           t.Errorf("%s: %s: mtime=%s, want %s", zt.Name, f.Name, ft, mtime)
+       if mt := f.ModTime(); mt.Format("01-02-06 15:04:05") != ft.Mtime {
+           t.Errorf("%s: %s: mtime=%s, want %s", zt.Name, f.Name, mt, ft.Mtime)
        }
    }

diff --git a/src/archive/zip/struct.go b/src/archive/zip/struct.go
index 0be210e8e7..aff4d4b665 100644
--- a/src/archive/zip/struct.go
+++ b/src/archive/zip/struct.go
@@ -160,7 +160,7 @@ func msDosTimeToTime(dosDate, dosTime uint16) time.Time {
        int(dosTime&0x1f*2),
        0, // nanoseconds

-       time.UTC,
+       time.Local,
    )
 }

@@ -168,7 +168,7 @@ func msDosTimeToTime(dosDate, dosTime uint16) time.Time {
 // The resolution is 2s.
 // See: http://msdn.microsoft.com/en-us/library/ms724274(v=VS.85).aspx
 func timeToMsDosTime(t time.Time) (fDate uint16, fTime uint16) {
-   t = t.In(time.UTC)
+   t = t.In(time.Local)
    fDate = uint16(t.Day() + int(t.Month())<<5 + (t.Year()-1980)<<9)
    fTime = uint16(t.Second()/2 + t.Minute()<<5 + t.Hour()<<11)
    return
bobjalex commented 7 years ago

I think your paraphrase of my opinion is correct. Since the "main" timestamps contain no time zone info, the main timestamp is interpreted as the stored time in the time zone where it is unzipped. E.g. "2016/01/02 15:04:05" decodes as "2016/01/02 15:04:05 JST" in Japan and as "2016/01/02 15:04:05 PST" in California. The main timestamps are not very useful in "global" setting -- which is one of the reasons why the various "extra" timestamps have evolved, such as the extended timestamp. When zip was invented and everyone was using DOS, not many of its users cared, but as the world has become more networked, users that export their zip files globally depend on the extended timestamp (or one of the other extra timestamps) instead.

(The other reason for the extra timestamps is accuracy -- the resolution of the main timestamp is just 2 seconds!)

So, I'm happy with your proposal to fully support extended timestamps. But do hope that there will be a way to store and retrieve the main timestamp as a local time. And the extended timestamp should of course always be UTC, independent of what is stored in the main timestamp.

Personally... I don't think there is really a need to ever store the main timestamp as UTC, except possibly for compatibility with Go's current archive/zip. Storing the main timestamp as local time is how most of the zip packages I know work, and folks that need a better time use the extended timestamp.

On Sat, May 27, 2017 at 7:16 AM, mattn notifications@github.com wrote:

As far as I can read your opinion, I guess that you mean below.

When zip timestamp is written as "2016/01/02 15:04:05", ft.ModTime() should be 2016/01/02 15:04:05 JST for japanese local.

Right? Then patch will be below.

diff --git a/src/archive/zip/reader_test.go b/src/archive/zip/reader_test.go index dfaae78436..8a09f897bd 100644--- a/src/archive/zip/reader_test.go+++ b/src/archive/zip/reader_test.go@@ -15,7 +15,6 @@ import ( "regexp" "strings" "testing"- "time" )

type ZipTest struct {@@ -369,13 +368,8 @@ func readTestFile(t testing.T, zt ZipTest, ft ZipTestFile, f File) { }

if ft.Mtime != "" {- mtime, err := time.Parse("01-02-06 15:04:05", ft.Mtime)- if err != nil {- t.Error(err)- return- }- if ft := f.ModTime(); !ft.Equal(mtime) {- t.Errorf("%s: %s: mtime=%s, want %s", zt.Name, f.Name, ft, mtime)+ if mt := f.ModTime(); mt.Format("01-02-06 15:04:05") != ft.Mtime {+ t.Errorf("%s: %s: mtime=%s, want %s", zt.Name, f.Name, mt, ft.Mtime) } } diff --git a/src/archive/zip/struct.go b/src/archive/zip/struct.go index 0be210e8e7..aff4d4b665 100644--- a/src/archive/zip/struct.go+++ b/src/archive/zip/struct.go@@ -160,7 +160,7 @@ func msDosTimeToTime(dosDate, dosTime uint16) time.Time { int(dosTime&0x1f*2), 0, // nanoseconds

  • time.UTC,+ time.Local, ) } @@ -168,7 +168,7 @@ func msDosTimeToTime(dosDate, dosTime uint16) time.Time { // The resolution is 2s. // See: http://msdn.microsoft.com/en-us/library/ms724274(v=VS.85).aspx func timeToMsDosTime(t time.Time) (fDate uint16, fTime uint16) {- t = t.In(time.UTC)+ t = t.In(time.Local) fDate = uint16(t.Day() + int(t.Month())<<5 + (t.Year()-1980)<<9) fTime = uint16(t.Second()/2 + t.Minute()<<5 + t.Hour()<<11) return

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-304454502, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNATsoPqMJX5WE4Tx3H-JvHEt54JbKks5r-DAggaJpZM4LPvG1 .

mattn commented 7 years ago

@ianlancetaylor @rsc @dsnet @bradfitz

@bobjalex suggest the change to store main timestamp as local time. Can we do this?

gopherbot commented 7 years ago

CL https://golang.org/cl/44394 mentions this issue.

dsnet commented 7 years ago

It's not clear to me that this is the right change.

Section 4.4.6 of the ZIP specification says:

The date and time are encoded in standard MS-DOS format. If input came from standard input, the date and time are those at which compression was started for this data. If encrypting the central directory and general purpose bit flag 13 is set indicating masking, the value stored in the Local Header will be zero.

However, it fails to specify what the timezone is for the format. Looking at the MSDN manual for DosDateTimeToFileTime also fails to specify a default timezone. It seems to be entirely up to the application to decide what timezone to use (which is an insane liberty allowed by ZIP; but such is the state of ZIP). Thus, the precedence to use time.Local is simply that "everyone else seems to be doing that".

If we could re-write the zip package from scratch, it may have been better to use the Local timezone instead of UTC, but we have already be using UTC as the default since Go 1. There are probably many users of the zip package that already rely on this behavior, and I would not want to break them for this change. If we made this change, in some ways it makes the situation worse since prior versions of Go would have been using UTC and newer versions of Go would be using Local, and a reader of a ZIP would not be able to tell which was which.

The current status quo is unfortunate, but I believe the proper fix is to add full support in both the reader and writer for extended timestamps, which are well-defined to be UTC. I wanted to add support for them in Go1.9, but I dropped the ball on this and did not get to it.

I propose the following API changes to zip.FileHeader:

bobjalex commented 7 years ago

Sounds good. Just don't make it impossible to write zip files with extended timestamp as per the spec (UTC), and a main timestamp that optionally might be different from default that is chosen by the team. The 1.8 early release zip package did make it impossible to have a local main timestamp without also forcing an illegal local extended timestamp -- the extended timestamp should never be local.

On Tue, May 30, 2017 at 11:20 AM, Joe Tsai notifications@github.com wrote:

It's not clear to me that this is the right change.

Section 4.4.6 of the ZIP specification says:

The date and time are encoded in standard MS-DOS format. If input came from standard input, the date and time are those at which compression was started for this data. If encrypting the central directory and general purpose bit flag 13 is set indicating masking, the value stored in the Local Header will be zero.

However, it fails to specify that what the timezone is for the format. Looking at the MSDN manual for DosDateTimeToFileTime https://msdn.microsoft.com/en-us/library/ms724247(v=VS.85).aspx also fails to specify a default timezone. It seems to be entirely up to the application to decide what timezone to use (which is an insane liberty allowed by ZIP; but such is the state of ZIP). Thus, the precedence to use time.Local is simply that "every else seems to be doing that".

If we could re-write the zip package from scratch, it may have been better to use the Local timezone instead of UTC, but we have already be using UTC as the default since Go 1. There are probably many users of the zip package that already rely on this behavior, and I would not want to break them for this change. If we made this change, in some ways it makes the situation worse since prior versions of Go would have been using UTC and newer versions of Go would be using Local, and a reader of a ZIP would not be able to tell which was which.

The current status quo is unfortunate, but I believe the proper fix is to add full support in both the reader and writer for extended timestamps, which are well-defined to be UTC. I wanted to add support for them in Go1.9, but I dropped the ball on this and did not get to it.

I propose the following API changes to zip.FileHeader:

  • Add the field Modified time.Time. We name it Modified since ModifiedTime, ModifiedDate, and ModTime are unfortunately already used by the legacy approach. The divergence in naming from tar.Header.ModTime is also unfortunate.
  • Mark ModTime, SetModTime, ModifiedTime, and ModifiedDate as deprecated.
  • Using SetModTime updates both the Modified and ModifiedTime/ ModifiedDate fields (still assuming the MS-DOS timestamps are UTC).
  • When reading an archive, we populate Modified with the most accurate timestamp we find (which extended timestamp variant we support is TBD).
  • When writing an archive, if the Modified is non-zero, then we use it to the populate the main time field and also emit an extended timestamp field (the exact variant we use is TBD).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-304963840, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAc6FArL025okpSyvMoM3ZWfEYRioks5r_F3bgaJpZM4LPvG1 .

bobjalex commented 7 years ago

I forgot to say please :)

BTW, I hacked such an option into the 1.8 version. Worked, but I'm sure it could be done more cleanly with a rewrite.

On Tue, May 30, 2017 at 3:56 PM, Bob Alexander bobjalex@gmail.com wrote:

Sounds good. Just don't make it impossible to write zip files with extended timestamp as per the spec (UTC), and a main timestamp that optionally might be different from default that is chosen by the team. The 1.8 early release zip package did make it impossible to have a local main timestamp without also forcing an illegal local extended timestamp -- the extended timestamp should never be local.

On Tue, May 30, 2017 at 11:20 AM, Joe Tsai notifications@github.com wrote:

It's not clear to me that this is the right change.

Section 4.4.6 of the ZIP specification says:

The date and time are encoded in standard MS-DOS format. If input came from standard input, the date and time are those at which compression was started for this data. If encrypting the central directory and general purpose bit flag 13 is set indicating masking, the value stored in the Local Header will be zero.

However, it fails to specify that what the timezone is for the format. Looking at the MSDN manual for DosDateTimeToFileTime https://msdn.microsoft.com/en-us/library/ms724247(v=VS.85).aspx also fails to specify a default timezone. It seems to be entirely up to the application to decide what timezone to use (which is an insane liberty allowed by ZIP; but such is the state of ZIP). Thus, the precedence to use time.Local is simply that "every else seems to be doing that".

If we could re-write the zip package from scratch, it may have been better to use the Local timezone instead of UTC, but we have already be using UTC as the default since Go 1. There are probably many users of the zip package that already rely on this behavior, and I would not want to break them for this change. If we made this change, in some ways it makes the situation worse since prior versions of Go would have been using UTC and newer versions of Go would be using Local, and a reader of a ZIP would not be able to tell which was which.

The current status quo is unfortunate, but I believe the proper fix is to add full support in both the reader and writer for extended timestamps, which are well-defined to be UTC. I wanted to add support for them in Go1.9, but I dropped the ball on this and did not get to it.

I propose the following API changes to zip.FileHeader:

  • Add the field Modified time.Time. We name it Modified since ModifiedTime, ModifiedDate, and ModTime are unfortunately already used by the legacy approach. The divergence in naming from tar.Header.ModTime is also unfortunate.
  • Mark ModTime, SetModTime, ModifiedTime, and ModifiedDate as deprecated.
  • Using SetModTime updates both the Modified and ModifiedTime/ ModifiedDate fields (still assuming the MS-DOS timestamps are UTC).
  • When reading an archive, we populate Modified with the most accurate timestamp we find (which extended timestamp variant we support is TBD).
  • When writing an archive, if the Modified is non-zero, then we use it to the populate the main time field and also emit an extended timestamp field (the exact variant we use is TBD).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-304963840, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAc6FArL025okpSyvMoM3ZWfEYRioks5r_F3bgaJpZM4LPvG1 .

mattn commented 7 years ago

@dsnet I have objection to add new APIs to fix this because it's not treated in default for http.FileSystem or webdav.FileSystem or something like FileSystem served from zip archive. For example, zipfs return zip.File#ModTime for FileSystem.

https://github.com/golang/tools/blob/5a22c00969ca85e2a020645ccaf02fffa89d95ca/godoc/vfs/zipfs/zipfs.go#L52

So the serve file from zip often hit the cache even though the zip archive are updated. As far as I know, many users using archive/zip package doesn't notice this issue. Or they put workaround to add time offset, or using external zip command to make zip file (it's me).

bobjalex commented 7 years ago

This can be done with the existing API working just as before, using the default of UTC main time field. Here is how I implemented the option of local or UTC by adding a couple of new functions for specifying the option:

In reader.go, the OpenReader function works as always and invokes the default of local time in the main time field. A new function OpenReaderLocal does the same but allows specification of an option of UTC or local in the main time field.

// OpenReader will open the Zip file specified by name and return a // ReadCloser which will write mod times in UTC. func OpenReader(name string) (*ReadCloser, error) { return OpenReaderLocal(name, false) }

// OpenReaderLocal will open the Zip file specified by name and return a // ReadCloser which will write mod times in UTC or local time, depending // on the value of the localTime argument. func OpenReaderLocal(name string, localTime bool) (*ReadCloser, error) { ... }

In struct.go, the FileInfoHeader works just as always and invokes the default of UTC time in the main time field. A new function FileInfoHeaderLocal does the same but allows specification of an option of UTC or local in the main time field.

// FileInfoHeader creates a partially-populated FileHeader from an // os.FileInfo. // Because os.FileInfo's Name method returns only the base name of // the file it describes, it may be necessary to modify the Name field // of the returned header to provide the full path name of the file. func FileInfoHeader(fi os.FileInfo) (*FileHeader, error) { return FileInfoHeaderLocal(fi, false) }

// FileInfoHeader creates a partially-populated FileHeader from an // os.FileInfo which will allow writing mod times in UTC or local time, // depending on the value of the localTime argument. // Because os.FileInfo's Name method returns only the base name of // the file it describes, it may be necessary to modify the Name field // of the returned header to provide the full path name of the file. func FileInfoHeaderLocal(fi os.FileInfo, localTime bool) (*FileHeader, error) { ... }

I have a working implementation of this. You're welcome to the code if you'd like. My implementation is a modification of the early 1.8 archive/zip (done for my personal use only).

On Tue, May 30, 2017 at 5:15 PM, mattn notifications@github.com wrote:

@dsnet https://github.com/dsnet I have objection to add new APIs to fix this because it's not treated in default for http.FileSystem or webdav.FileSystem or something like FileSystem served from zip archive. For example, zipfs return zip.File#ModTime for FileSystem.

https://github.com/golang/tools/blob/5a22c00969ca85e2a020645ccaf02f ffa89d95ca/godoc/vfs/zipfs/zipfs.go#L52

So the serve file from zip often hit the cache even though the zip archive are updated. As far as I know, many users using archive/zip package doesn't notice this issue. Or they put workaround to add time offset, or using external zip command to make zip file (it's me).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-305044886, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAeTuDKTXu5Lq1BZOrsldAM8O1hPFks5r_LE0gaJpZM4LPvG1 .

bobjalex commented 7 years ago

For zip filesystems, the main time field is pretty questionable whether UTC or local. It has only 2-second resolution/accuracy -- not really suitable for modern filesystems. Probably want to always use the extended tmestamp.

On Tue, May 30, 2017 at 7:24 PM, Bob Alexander bobjalex@gmail.com wrote:

This can be done with the existing API working just as before, using the default of UTC main time field. Here is how I implemented the option of local or UTC by adding a couple of new functions for specifying the option:

In reader.go, the OpenReader function works as always and invokes the default of local time in the main time field. A new function OpenReaderLocal does the same but allows specification of an option of UTC or local in the main time field.

// OpenReader will open the Zip file specified by name and return a // ReadCloser which will write mod times in UTC. func OpenReader(name string) (*ReadCloser, error) { return OpenReaderLocal(name, false) }

// OpenReaderLocal will open the Zip file specified by name and return a // ReadCloser which will write mod times in UTC or local time, depending // on the value of the localTime argument. func OpenReaderLocal(name string, localTime bool) (*ReadCloser, error) { ... }

In struct.go, the FileInfoHeader works just as always and invokes the default of UTC time in the main time field. A new function FileInfoHeaderLocal does the same but allows specification of an option of UTC or local in the main time field.

// FileInfoHeader creates a partially-populated FileHeader from an // os.FileInfo. // Because os.FileInfo's Name method returns only the base name of // the file it describes, it may be necessary to modify the Name field // of the returned header to provide the full path name of the file. func FileInfoHeader(fi os.FileInfo) (*FileHeader, error) { return FileInfoHeaderLocal(fi, false) }

// FileInfoHeader creates a partially-populated FileHeader from an // os.FileInfo which will allow writing mod times in UTC or local time, // depending on the value of the localTime argument. // Because os.FileInfo's Name method returns only the base name of // the file it describes, it may be necessary to modify the Name field // of the returned header to provide the full path name of the file. func FileInfoHeaderLocal(fi os.FileInfo, localTime bool) (*FileHeader, error) { ... }

I have a working implementation of this. You're welcome to the code if you'd like. My implementation is a modification of the early 1.8 archive/zip (done for my personal use only).

On Tue, May 30, 2017 at 5:15 PM, mattn notifications@github.com wrote:

@dsnet https://github.com/dsnet I have objection to add new APIs to fix this because it's not treated in default for http.FileSystem or webdav.FileSystem or something like FileSystem served from zip archive. For example, zipfs return zip.File#ModTime for FileSystem.

https://github.com/golang/tools/blob/5a22c00969ca85e2a020645 ccaf02fffa89d95ca/godoc/vfs/zipfs/zipfs.go#L52

So the serve file from zip often hit the cache even though the zip archive are updated. As far as I know, many users using archive/zip package doesn't notice this issue. Or they put workaround to add time offset, or using external zip command to make zip file (it's me).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-305044886, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAeTuDKTXu5Lq1BZOrsldAM8O1hPFks5r_LE0gaJpZM4LPvG1 .

bobjalex commented 7 years ago

Joe, that proposal seems complicated compared to the API supplementation that I did. With my API, nothing changes at all for users of previous versions. But if they want the main time stamp to be local instead of UTC, they use the 2 new functions (which are almost the same as the old ones) instead of the corresponding old ones.

Your suggestion allows any time at all in the main time field, which is a good idea (mine just specifies UTC or local). That can be done in my approach by simply replacing my boolean arguments that specify local or UTC with time.Time arguments that specify exactly what goes into the main field. This approach just seems so simple, and creates no ripples for users of the previous version.

On Tue, May 30, 2017 at 11:20 AM, Joe Tsai notifications@github.com wrote:

It's not clear to me that this is the right change.

Section 4.4.6 of the ZIP specification says:

The date and time are encoded in standard MS-DOS format. If input came from standard input, the date and time are those at which compression was started for this data. If encrypting the central directory and general purpose bit flag 13 is set indicating masking, the value stored in the Local Header will be zero.

However, it fails to specify that what the timezone is for the format. Looking at the MSDN manual for DosDateTimeToFileTime https://msdn.microsoft.com/en-us/library/ms724247(v=VS.85).aspx also fails to specify a default timezone. It seems to be entirely up to the application to decide what timezone to use (which is an insane liberty allowed by ZIP; but such is the state of ZIP). Thus, the precedence to use time.Local is simply that "every else seems to be doing that".

If we could re-write the zip package from scratch, it may have been better to use the Local timezone instead of UTC, but we have already be using UTC as the default since Go 1. There are probably many users of the zip package that already rely on this behavior, and I would not want to break them for this change. If we made this change, in some ways it makes the situation worse since prior versions of Go would have been using UTC and newer versions of Go would be using Local, and a reader of a ZIP would not be able to tell which was which.

The current status quo is unfortunate, but I believe the proper fix is to add full support in both the reader and writer for extended timestamps, which are well-defined to be UTC. I wanted to add support for them in Go1.9, but I dropped the ball on this and did not get to it.

I propose the following API changes to zip.FileHeader:

  • Add the field Modified time.Time. We name it Modified since ModifiedTime, ModifiedDate, and ModTime are unfortunately already used by the legacy approach. The divergence in naming from tar.Header.ModTime is also unfortunate.
  • Mark ModTime, SetModTime, ModifiedTime, and ModifiedDate as deprecated.
  • Using SetModTime updates both the Modified and ModifiedTime/ ModifiedDate fields (still assuming the MS-DOS timestamps are UTC).
  • When reading an archive, we populate Modified with the most accurate timestamp we find (which extended timestamp variant we support is TBD).
  • When writing an archive, if the Modified is non-zero, then we use it to the populate the main time field and also emit an extended timestamp field (the exact variant we use is TBD).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-304963840, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAc6FArL025okpSyvMoM3ZWfEYRioks5r_F3bgaJpZM4LPvG1 .

mattn commented 7 years ago

@bobjalex Could you please create new CL?

mattn commented 7 years ago

@bobjalex If you have diff, could you please send CL. Or show me your changes if possible. Thanks.

bobjalex commented 7 years ago

OK, here are my changes. Disclaimer: I'm doing this at home in a Windows-only environment, and currently have no access to Linux or a "professional" tools environment. And I'm not exactly sure what CL format is. So... I did a diff using Cygwin and options -cs -- attached. Hope this is easy for you to work with.

Disclaimer 2: looks like I also did some fiddling with supporting more of the extra datetime fields -- largely untested but doesn't seem to crash

This code has been kept private to me, and hasn't been given to anyone except, now, you folks. Feel free to use it as you wish.

On Wed, Jun 14, 2017 at 10:45 PM, mattn notifications@github.com wrote:

@bobjalex https://github.com/bobjalex If you have diff, could you please send CL. Or show me your changes if possible. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-308634816, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAaUcKFDhWCZwaSWqQiijOILQkrI-ks5sEMURgaJpZM4LPvG1 .

C:\Users\Bob>diff -cs C:\GoSrc\go\src\archive\zip C:\GoLib\src\bobalex\zip Only in C:\GoSrc\go\src\archive\zip: example_test.go diff -cs "C:\GoSrc\go\src\archive\zip/reader.go" "C:\GoLib\src\bobalex\zip/reader.go" *** "C:\GoSrc\go\src\archive\zip/reader.go" Thu Dec 1 13:16:26 2016 --- "C:\GoLib\src\bobalex\zip/reader.go" Tue May 30 19:26:40 2017

* 13,18 ** --- 13,19 ---- "hash/crc32" "io" "os"

! // OpenReader will open the Zip file specified by name and return a ReadCloser. func OpenReader(name string) (*ReadCloser, error) { f, err := os.Open(name) if err != nil { return nil, err --- 47,63 ---- return f.Flags&0x8 != 0 }

! // OpenReader will open the Zip file specified by name and return a ReadCloser ! // which will write mod times in UTC. func OpenReader(name string) (*ReadCloser, error) {

! time.UTC, ) }

// timeToMsDosTime converts a time.Time to an MS-DOS date and time. // The resolution is 2s. // See: http://msdn.microsoft.com/en-us/library/ms724274(v=VS.85).aspx ! func timeToMsDosTime(t time.Time) (fDate uint16, fTime uint16) { ! t = t.In(time.UTC) fDate = uint16(t.Day() + int(t.Month())<<5 + (t.Year()-1980)<<9) fTime = uint16(t.Second()/2 + t.Minute()<<5 + t.Hour()<<11) return }

! // ModTime returns the modification time in UTC. // The resolution is 2s. func (h *FileHeader) ModTime() time.Time { ! return msDosTimeToTime(h.ModifiedDate, h.ModifiedTime) }

! // SetModTime sets the ModifiedTime and ModifiedDate fields to the given time in UTC. // The resolution is 2s. func (h *FileHeader) SetModTime(t time.Time) { ! h.ModifiedDate, h.ModifiedTime = timeToMsDosTime(t) }

const ( --- 190,226 ---- int(dosTime&0x1f*2), 0, // nanoseconds

! loc, ) }

// timeToMsDosTime converts a time.Time to an MS-DOS date and time. // The resolution is 2s. // See: http://msdn.microsoft.com/en-us/library/ms724274(v=VS.85).aspx ! func timeToMsDosTime(t time.Time, localTime bool) (fDate uint16, fTime uint16) { ! if localTime { ! t = t.Local() ! } else { ! t = t.UTC() ! } fDate = uint16(t.Day() + int(t.Month())<<5 + (t.Year()-1980)<<9) fTime = uint16(t.Second()/2 + t.Minute()<<5 + t.Hour()<<11) return }

! // ModTime returns the modification time in UTC or local, ! // depending on h.localTime. // The resolution is 2s. func (h *FileHeader) ModTime() time.Time { ! return msDosTimeToTime(h.ModifiedDate, h.ModifiedTime, h.localTime) }

! // SetModTime sets the ModifiedTime and ModifiedDate fields to the given time ! // in UTC. // The resolution is 2s. func (h *FileHeader) SetModTime(t time.Time) { ! h.modTime = t ! h.ModifiedDate, h.ModifiedTime = timeToMsDosTime(t, h.localTime) }

const ( Only in C:\GoSrc\go\src\archive\zip: testdata diff -cs "C:\GoSrc\go\src\archive\zip/writer.go" "C:\GoLib\src\bobalex\zip/writer.go" *** "C:\GoSrc\go\src\archive\zip/writer.go" Thu Dec 1 13:16:26 2016 --- "C:\GoLib\src\bobalex\zip/writer.go" Mon Dec 19 12:57:09 2016

* 22,27 ** --- 22,31 ---- last *fileWriter closed bool compressors map[uint16]Compressor +

! if records > uint16max || size > uint32max || offset > uint32max { var buf [directory64EndLen + directory64LocLen]byte b := writeBuf(buf[:])

--- 144,154 ---- size := uint64(end - start) offset := uint64(start)

! if f := w.testHookCloseSizeOffset; f != nil { ! f(size, offset) ! } ! ! if records >= uint16max || size >= uint32max || offset >= uint32max { var buf [directory64EndLen + directory64LocLen]byte b := writeBuf(buf[:])

* 376,381 ** --- 397,407 ----

type writeBuf []byte

mattn commented 7 years ago

@bobjalex Thank you very much. I will make new CL based on your changes. @dsnet Do you mind that I'll put thanks to bobjalx into the commit comment? (Because this is not my idea)

bobjalex commented 7 years ago

Do you mind that I'll put thanks to bobjalx into the commit comment?'

It would be an honor.

But it's bobjalex :) (Google lD "ralex", from when I contracted there a few times)

On Thu, Jun 15, 2017 at 9:47 AM, mattn notifications@github.com wrote:

@bobjalex https://github.com/bobjalex Thank you very much. I will make new CL based on your changes. @dsnet https://github.com/dsnet Do you mind that I'll put thanks to bobjalx into the commit comment?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-308799657, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNASJ7ZCkYybGR_AgeRxg5mzxyIA9cks5sEWAzgaJpZM4LPvG1 .

dsnet commented 7 years ago

If one of you sends a diff, I can review it. Pleaes be aware that this will not make Go1.9 as the freeze happened long ago, but we can definitely try to get a proper fix in for Go1.10

dsnet commented 7 years ago

@bobjalex, you're GitHub user doesn't seem to be in the system for having signed the CLA. I ain't a lawyer, I'm not sure how this works if parts of the code that @mattn contributes is actually written by you.

rsc commented 7 years ago

Right, please don't use code posted to Github. Only use code posted to Gerrit by its original author. If someone can't complete a CLA and post code to Gerrit for whatever reason, that's OK. We can almost always independently write the code ourselves instead. (And when that's not true we just live with not having the code.) Thanks.

mattn commented 7 years ago

@dsnet @rsc Agree with you.

bobjalex commented 7 years ago

I'm not a lawyer either -- just a guy coding at home on my personal Windows system, not using github or any source management system. I'm currently semi-retired and am not employed. Other than what I have sent to you, the code exists only on non-networked computers in my home. I sent my code to y'all to demonstrate a technique that I like for getting the desired dates into the various zip date fields, following discussions with golang team members. The code I sent is a modification of Google's source code for a previous zip package version. I offer my code for you to use in any way you would like, verbatim or not. I don't know much about possible legal entanglements.

Whew -- given all that, I hope my code can be helpful to you :)

On Thu, Jun 15, 2017 at 5:52 PM, mattn notifications@github.com wrote:

@dsnet https://github.com/dsnet @rsc https://github.com/rsc Agree with you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-308903380, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNASZ6kuYkxZ5J3ivAoc4ZLu1J9RqGks5sEdG4gaJpZM4LPvG1 .

rsc commented 7 years ago

Hi @bobjalex. I do appreciate your willingness to help. I checked with our lawyers and it's OK for us to accept the diff here, since you said we can use it, provided you complete a contributor CLA as discussed in https://golang.org/doc/contribute.html#cla (the "Contributor License Agreement" section). Can you do that?

Thanks very much.

bobjalex commented 7 years ago

Thanks Russ (& Co.) I'll have a look at the CLA and get back -- not expecting any problems.

On Fri, Jun 16, 2017 at 11:00 AM, Russ Cox notifications@github.com wrote:

Hi @bobjalex https://github.com/bobjalex. I do appreciate your willingness to help. I checked with our lawyers and it's OK for us to accept the diff here, since you said we can use it, provided you complete a contributor CLA as discussed in https://golang.org/doc/contribute.html#cla (the "Contributor License Agreement" section). Can you do that?

Thanks very much.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-309093505, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAbg_0ls3GYtFEoNzVN5yOj7X3NkWks5sEsKjgaJpZM4LPvG1 .

bobjalex commented 7 years ago

I submitted my contributor CLA (finally, sorry it took so long).

On Fri, Jun 16, 2017 at 1:54 PM, Bob Alexander bobjalex@gmail.com wrote:

Thanks Russ (& Co.) I'll have a look at the CLA and get back -- not expecting any problems.

On Fri, Jun 16, 2017 at 11:00 AM, Russ Cox notifications@github.com wrote:

Hi @bobjalex https://github.com/bobjalex. I do appreciate your willingness to help. I checked with our lawyers and it's OK for us to accept the diff here, since you said we can use it, provided you complete a contributor CLA as discussed in https://golang.org/doc/contrib ute.html#cla (the "Contributor License Agreement" section). Can you do that?

Thanks very much.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/18359#issuecomment-309093505, or mute the thread https://github.com/notifications/unsubscribe-auth/AFYNAbg_0ls3GYtFEoNzVN5yOj7X3NkWks5sEsKjgaJpZM4LPvG1 .

bradfitz commented 7 years ago

I confirm that we have the CLA on file for @bobjalex.

rsc commented 7 years ago

OK, thanks @bobjalex. @dsnet, can you look at this for Go 1.10?

mattn commented 7 years ago

@bobjalex something problems for create CL?

gopherbot commented 7 years ago

Change https://golang.org/cl/74970 mentions this issue: archive/zip: add FileHeader.Modified field

dsnet commented 7 years ago

@bobjalex, take a look at CL/74970 to see if it suites your needs. It maintains backwards compatibility where UTC is typically the timezone used in the legacy timestamp fields.

However, since the FileHeader.Modified field is newly added, you can always set it directly to control the timezone used for the date.

fh, _ := zip.FileInfoHeader(fi) // This uses UTC for Modified to be backwards compatible
fh.Modified = dh.Modified.In(time.Local)
mattn commented 7 years ago

@dsnet Thank you for working this. I'm totally agreed to your CL. One question, how do you think about zipfs issue? We have to convert Modified for each items in the zip file?

https://github.com/golang/go/issues/18359#issuecomment-305044886

dsnet commented 7 years ago

Why would this be an issue with zipfs? The time returned will be returned as UTC as before, and will be an accurate point in time (assuming the underlying zip has extended timestamps).

Rather, I should ask, why would the user care for the original timezone of the underlying zip file?

Essentially, my CL should not change the behavior of zipFI.ModTime in any way (other than being correct if zip has extended timestamps).

mattn commented 7 years ago

This is a file that archived using zip command.

foo.zip

file timestamp of write.go is 2017/11/02 10:58.

image

and file write.go in the zip archive have same timestamp.

image

Using CL74970 and following code.

package main

import (
    "archive/zip"
    "io/ioutil"
    "log"
    "os"
)

func ZipFile(writer *zip.Writer, zipPath string, targetFile string) error {
    f, err := os.Open(targetFile)
    if err != nil {
        return err
    }
    defer f.Close()

    body, err := ioutil.ReadAll(f)

    if err != nil {
        return err
    }

    info, err := os.Stat(targetFile)
    if err != nil {
        return err
    }

    header, _ := zip.FileInfoHeader(info)
    header.Name = zipPath

    zf, err := writer.CreateHeader(header)
    if err != nil {
        return err
    }

    if _, err := zf.Write(body); err != nil {
        return err
    }
    return nil
}

func main() {
    f, err := os.Create("my.zip")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    zw := zip.NewWriter(f)

    if ferr := ZipFile(zw, "write.go", "write.go"); ferr != nil {
        panic(ferr)
    }

    if ferr := zw.Close(); ferr != nil {
        panic(ferr)
    }
}

my.zip

image

the timestamp of write.go in the zip archive is not same.

dsnet commented 7 years ago

Just to confirm, this was the same behavior before CL74970, right? If so, this is what I expect since Windows doesn't understand the extended timestamps.

If you do:

    header, _ := zip.FileInfoHeader(info)
    header.Name = zipPath
    header.Modified = header.Modified.In(time.Local)

Does that fix it?

mattn commented 7 years ago

Does that fix it?

Yes. it fixed correctly. Does it require to call header.Modifierd.In to be handled in our timestamp for Reader and Writer both?