camac / dora

Domino On-Disk Repository Assistant
http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&name=Dora
Apache License 2.0
8 stars 8 forks source link

Can't add a file (Using SourceTree) #20

Closed majkilde closed 6 years ago

majkilde commented 11 years ago

When I tri to add a file, I get this message:

git -c diff.mnemonicprefix=false -c core.quotepath=false add -f -- app/AppProperties/database.properties Unknown option -C:/Program Files (x86)/Git/xsltp

error: external filter c:/Users/Jakob/bin/xsltproc xsl/DXLClean.xsl - failed 3 error: external filter c:/Users/Jakob/bin/xsltproc xsl/DXLClean.xsl - failed fatal: app/AppProperties/database.properties: clean filter 'dxlmetadata' failed

Do you know what is wrong?

camac commented 11 years ago

Hi Jakob, Sourcetree will fail like this (and also won't show you the 'diff' in the right panel) if it can't run the filter associated with the file.

Is this a new NSF? Judging from the filename 'database.properties' I think it may be, because I have had this problem as well. There is a 'Known Issue' in that if the File is empty/blank/not well-formed xml then the filter will fail. When you create a new nsf, the database.properties file and a couple of others (see Readme near the bottom) will be blank until you deliberately save them for the first time. So to get around this you just need to save each of these files one time to 'initialise' them properly. After this one-time fix you should not get the problem again for these files.

If the problem is not related to the blank file then some things to check are:

oysteinjaren commented 11 years ago

I have almost the same problem. I am currently using Git GUI. There is a metadata file which has some changes. This is what the diff looks like without Dora filtering (correct):

image

After installing Dora, the file is still listed as changed in Git GUI, but the diff window shows nothing; it's blank. Also, if I run "git status" in bash (or try to stage the file), I get this error:

Unknown option -C:/Users/E6098/AppData/Local/Programs/Git/bin/x☻
error: external filter n:/bin/xsltproc xsl/DXLClean.xsl - failed 3
error: external filter n:/bin/xsltproc xsl/DXLClean.xsl - failed
fatal: DevMalKonsolidertHoved/Resources/Images/_article_top_shade.gif.metadata: clean filter 'dxlmetadata' failed

(Yes, that's a smiley face at the end of the first line...)

I tried removing the dashes at the end of the first two lines that Dora inserts in the gitconfig file. (What are those for?)

image

After removing those, I can run "git status" again. The metadata file is listed as modified, but the diff is definitely wrong. It looks like the entire content of the file has been removed, which is certainly not right:

image

Testing the XSL on the file using "xsltproc.exe xsl/DXLClean.xsl _article_top_shade.gif.metadata gives me the following:

<?xml version="1.0"?>
<note xmlns="http://www.lotus.com/dxl" class="form" maintenanceversion="3.0">
  <item name="$TITLE">
    <text>_article_top_shade.gif</text>
  </item>
  <item name="$FlagsExt" sign="true">
    <text>D</text>
  </item>
  <item name="$ImagesWide" sign="true">
    <number>1</number>
  </item>
  <item name="$ImagesHigh" sign="true">
    <number>1</number>
  </item>
  <item name="$ImagesColorize" sign="true">
    <number>0</number>
  </item>
  <item name="$MimeType" sign="true">
    <text>image/gif</text>
  </item>
  <item name="$FileModDT" sign="true">
    <datetime dst="true">20090423T112004,84+02</datetime>
  </item>
  <item name="$ImageNames" sign="true">
    <text>_article_top_shade.gif</text>
  </item>
  <item name="$Comment" sign="true">
    <text>nytt design / bekk</text>
  </item>
  <item name="$Flags">
    <text>34CiQ$</text>
  </item>
  <item name="$EditFilePath">
    <text>C:\Users\E6098\AppData\Local\Temp\notes87944B\~_article_top_shade.gif</text>
  </item>
</note>

This looks sensible to me, as all the unimportant information has been removed.

Do you have some idea as to what the problem could be?

camac commented 11 years ago

Hi Øystein (is that your first name? please correct me if wrong) thanks for the bug report and sorry that installation of dora didn't go smoothly... I appreciate the feedback, and I hope I can help resolve the problem.

I'm sorry to say I don't immediately know what the problem is, but I am keen to figure it out but might need some of your help to get there.

Could you please tell me what version of git and git gui you are running? (Help -> About git gui). I will try to replicate your set up to see if I can get the same error.

Are you running git bash from the windows command prompt? or from the MINGW32 prompt? if you aren't sure, try running a dir command it will say command not found if you are in MINGW32 version.

The smiley face in the filepath is strange, have you seen that before in any other case? Can you have a look in that folder c:\users\....\git\bin\ and tell me what's in there (in particular anything starting with x)? I don't have that folder, is this where your git installation is?

Some questions that might help me: Has dora been able to filter any files successfully? ie would this problem be isolated to this file? or is this simply the first file that has tried to be filtered and it has failed? (indicating this is a problem in general with dora setup)

When you ran Install.pl (or if you run it again) what does it say for target Directories for Binaries and resources? targetdirs

It is good to see xsltproc is filtering properly, I agree the output of that test looks normal. I'd like to double check the location of xsltproc as this can sometimes be a problem. What does it say if you run which xsltproc from git bash?

If there is anything else about your environment you can tell me that would be great as well. operating system, any deviation to the dora installation process, any access restrictions. Did you make any modifications to your PATH variables?

In regards to the hyphen, it is necessary to keep it there as it signifies to xsltproc to read from the standard input instead of from a file. Git passes the file to be filtered through to xsltproc as standard input.

Thanks for your assistance I hope I can get this figured out for you

Cameron

oysteinjaren commented 11 years ago

Hi. Sorry I couldn't get back to you sooner. It's been a little crazy here. And yes, you got my name right. =)

Git GUI reports that the Git version is 1.8.1.msysgit.1, and the Git GUI version is 0.17.GITGUI. Git Bash reports Git version 1.8.1.2-preview20130201. I can try upgrading to a newer version if you think that might help.

The dir command cannot be found, so I guess it's the MINGW32 version.

I have not seen the smiley face in any other case. The folder c:\users....\git is where my Git installation is. There are lots of files in c:\users....\git\bin, mostly executables and dll files. There is just one starting with x: xargs.exe.

However, I just realized that the home directory on this computer is on a file share (N:). I thought this might be causing problems, so I uninstalled Dora, changed the $homeDir variable in Install.pl and dora.pl from $ENV{"HOME"} to "C:/Users/E6098" and added "C:/Users/E6098/bin" to the PATH in ...\Git\etc\profile. Sadly this did not help, but the error message changed slightly:

Unknown option -C:/Users/E6098/AppData/Local/Programs/Git/xsltp☻
error: external filter C:/Users/E6098/bin/xsltproc xsl/DXLClean.xsl - failed 3
error: external filter C:/Users/E6098/bin/xsltproc xsl/DXLClean.xsl - failed
fatal: Aktuelt/Pages/helsesjekk.page: clean filter 'dxlmetadata' failed

Notice the end of the first line now reads /Git/xsltp☻ instead of /Git/bin/x☻, so the file it's trying to get at must be xsltproc.exe. But there is no file beginning with "xsltp" in C:/Users/E6098/AppData/Local/Programs/Git or C:/Users/E6098/AppData/Local/Programs/Git/bin

The problem appears to affect all the files that are supposed to be filtered. Trying to stage any such file in Git GUI always gives me the error. So it seems to be a general problem with the Dora setup.

Before I moved the Dora installation, the target directories were:

Binaries : /n/bin
Resources: /n/dora

Now they are:

Binaries : C:/Users/E6098/bin
Resources: C:/Users/E6098/dora

I never tried to run which xsltproc before I moved the installation, but now at least it outputs /c/Users/E6098/bin/xsltproc, which looks right to me.

camac commented 11 years ago

Hi Øystein (and Jakob!) Ok I don't have the answer yet but you have given me some more info to go on and I think I am getting closer to finding out how to fix it.

Your git versions seem fine to me, I don't think there would be any need to change that.

I didn't realize this before in both yours and Jakob's case - Git is successful in calling xsltproc, however for some reason the arguments it is passing to it is causing it to fail. So at this point the location of xsltproc is not a problem but something to do with the arguments being supplied to it. The evidence is in the error message:

Unknown option -C:/Users/E6098/AppData/Local/Programs/Git/xsltp☻
error: external filter C:/Users/E6098/bin/xsltproc xsl/DXLClean.xsl - failed 3
error: external filter C:/Users/E6098/bin/xsltproc xsl/DXLClean.xsl - failed
fatal: Aktuelt/Pages/helsesjekk.page: clean filter 'dxlmetadata' failed

The first line Unknown option ... is directly echoed from xsltproc, and the xsltproc return code 3, for unknown option is shown in the second line. You can see xsltproc error codes here: http://xmlsoft.org/XSLT/xsltproc.html

You can see this behaviour from xsltproc by running some bogus command like xsltproc -notreallyanoption you should see Unknown option -notreallyanoption and then if with your next command you echo the return code echo $? it should show 3

So basically it is an error message from xsltproc saying that it has been given a bad option, and the next step is to see if we can see what command git is calling, and why it might be including that C:/Users/... filepath as if it was an option.

For now lets focus on getting it to work with git bash and then after that will see if there is any further problem with git gui. When you get the spare time to do so, could you please do the following?

  1. Send me the output of git config --list executed from within your repository.
  2. run GIT_TRACE=1 git add --dry-run <FILENAME> where <FILENAME> is the name of a file that you know should be able to be filtered. e.g. xsltproc xsl/DXLClean.xsl <FILENAME> works as expected. the --dry-run option is just there so it won't actually stage the file but it doesn't matter if it's not there. The GIT_TRACE=1 option should show us what commands git is issuing. e.g. for me I can see trace: run_command: 'c:/Users/Cameron/bin/xsltproc xsl/DXLClean.xsl - I am hoping we see something different there that can give us a clue.

Thanks for you assistance I hope I can help get to a solution soon, let me know if you have any questions about the above

oysteinjaren commented 11 years ago

Here is the output of git config --list (with my e-mail address and origin url anonymized):

$ git config --list
core.symlinks=false
core.autocrlf=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
pack.packsizelimit=2g
help.format=html
http.sslcainfo=/bin/curl-ca-bundle.crt
sendemail.smtpserver=/bin/msmtp.exe
diff.astextplain.textconv=astextplain
rebase.autosquash=true
user.name=Øystein Samuelsen
user.email=<hidden>
gui.recentrepo=C:/therepo
merge.tool=kdiff3
diff.tool=kdiff3
diff.guitool=kdiff3
core.editor="C:/Users/e6098/Downloads/GitExtensions246Mono/GitExtensions/GitExtensions.exe" fileeditor
core.autocrlf=true
credential.helper=!"C:/Users/e6098/Downloads/GitExtensions246Mono/GitExtensions/GitCredentialWinStore/git-credential-winstore.exe"
alias.lol=log --graph --decorate --pretty=oneline --abbrev-commit --all
mergetool.kdiff3.path=C:/Users/E6098/bin/PortableApps/KDiff3/kdiff3.exe
difftool.kdiff3.path=C:/Users/E6098/bin/PortableApps/KDiff3/kdiff3.exe
difftool.kdiff3.cmd="C:/Users/E6098/bin/PortableApps/KDiff3/kdiff3.exe" "$LOCAL" "$REMOTE"
core.repositoryformatversion=0
core.filemode=false
core.bare=false
core.logallrefupdates=true
core.symlinks=false
core.ignorecase=true
core.hidedotfiles=dotGitOnly
remote.origin.url=<hidden>
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
gui.wmstate=zoomed
gui.geometry=443x321+200+200 456 709
branch.develop.remote=origin
branch.develop.merge=refs/heads/develop
branch.popup.remote=origin
branch.popup.merge=refs/heads/popup
filter.dxlmetadata.clean=n:/bin/xsltproc xsl/DXLClean.xsl -
filter.dxlmetadata.smudge=n:/bin/xsltproc xsl/DXLSmudge.xsl -
filter.dxlmetadata.required=true

Here is the output of $ GIT_TRACE=1 git add --dry-run testfile.form:

$ GIT_TRACE=1 git add --dry-run testfile.form
trace: built-in: git 'add' '--dry-run' 'testfile.form'
trace: run_command: 'n:/bin/xsltproc xsl/DXLClean.xsl -'
Unknown option -C:/Users/E6098/AppData/Local/Programs/Git/bin/x☻
error: cannot feed the input to external filter n:/bin/xsltproc xsl/DXLClean.xsl -
error: external filter n:/bin/xsltproc xsl/DXLClean.xsl - failed 3
error: external filter n:/bin/xsltproc xsl/DXLClean.xsl - failed
fatal: testfile.form: clean filter 'dxlmetadata' failed

By the way, I had an issue with Dora reporting the DXL filter as "Not installed" after I moved the Dora installation to my hard-coded location on the C drive, so I moved the installation back to the original location on the N fileshare, which is my actual home directory. It does not seem to make any difference in relation to our little problem.

I also started using Git Extensions in place of Git GUI (because it is much nicer to work with), but the problem is much the same there, so I think we can safely ignore the GUI aspect and stick to the command line.

camac commented 11 years ago

Thanks for that, I can see that the command it is issuing is correct: n:/bin/xsltproc xsl/DXLClean.xsl - the new clue is error: cannot feed the input to external filter n:/bin/xsltproc xsl/DXLClean.xsl - This makes me think the problem is to do with how git is passing the file contents to xsltproc through STDIN. I can think of 2 tests that might help isolate the problem. Firstly I'd like to test that xsltproc does not have a problem receiving the file contents through STDIN. To do this can you run a similar command to the one we used to test the filter before, but this time instead of supplying the filename, we will pipe in the contents from STDIN cat testfile.form | xsltproc xsl/DXLClean.xsl - The output from this should be the same as xsltproc xsl/DXLClean.xsl testfile.form If there is no error here, then xsltproc has no problem with the STDIN method, and the problem is likely to involve git itself. Next I would like to test if the problem is between git and any filter, not just xsltproc. To do this, we can set the dxlmetadata.clean filter to be cat which will not really perform any modification to the contents, just repeat it out again, but we will be testing passing the file contents to a very simple filter. To set the filter to cat git config filter.dxlmetadata.clean cat and then just check it worked git config --list should now show filter.dxlmetadata.clean=cat Then perform the test git add command GIT_TRACE=1 git add --dry-run testfile.form See if we get a similar error to xsltproc filter. If the cat filter runs ok, then we know it has something to to with xsltproc.

I was also wondering whether you and Jakob may have any regional settings that may affect text file encoding? What does it say in your Domino Designer preferences under: File -> Preferences, then General -> Workspace Under 'Text file encoding' Mine says 'Default (Cp1252)' Also what are your windows regional settings? what would be different to English settings?

Regarding Install location, yeah I think you are right I don't think it has any effect in relation to this problem. To explain the 'not installed', it would be because the filter that was previously installed no longer matched the new target directory.

I have never looked at Git Extensions actually, I was using git bash before and then have moved to SourceTree. It is much easier to inspect history with the GUI I find. I agree with you I think this problem is independent of GUI, as I think Jakob had the same problem with SourceTree.

Once again, thanks for your patience and let me know if you have any other questions!

oysteinjaren commented 11 years ago

Thank you very much for your suggestions.

Both xsltproc xsl/DXLClean.xsl testfile.form and cat testfile.form | xsltproc xsl/DXLClean.xsl - yield the same correct output. However, I discovered that if I specify the path to xsltproc like this: cat testfile.form | /n/bin/xsltproc xsl/DXLClean.xsl -, it fails with a familiar error message:

$ cat testfile.form | /n/bin/xsltproc xsl/DXLClean.xsl -
Unknown option -C:/Users/E6098/AppData/Local/Programs/Git/bin/x?
Usage: n:\bin\xsltproc.exe [options] stylesheet file [file ...]
    Options:
    ....

Though doing the same thing with the first command /n/bin/xsltproc xsl/DXLClean.xsl testfile.form works fine. Also, specifying the path differently like so: cat testfile.form | n:/bin/xsltproc.exe xsl/DXLClean.xsl - works fine.

Using cat as filter seems fine:

$ GIT_TRACE=1 git add --dry-run testfile.form
trace: built-in: git 'add' '--dry-run' 'testfile.form'
trace: run_command: 'cat'
add 'testfile.form'

Text file encoding in Domino Designer is the same as yours: "Default (Cp1252)". My regional settings are Norwegian. Something that has caused problems for me before with Norwegian regional settings is the decimal marker, which is "," rather than ".". But I doubt that has any importance here. Also the list delimiter is ";" instead of ",".

Now, because cat testfile.form | n:/bin/xsltproc.exe xsl/DXLClean.xsl - worked, I tried adding ".exe" to the xsltproc command in git/config, like so:

clean = n:/bin/xsltproc.exe xsl/DXLClean.xsl -
smudge = n:/bin/xsltproc.exe xsl/DXLSmudge.xsl -
required = true

Running $ GIT_TRACE=1 git add --dry-run testfile.form after this yields:

$ GIT_TRACE=1 git add --dry-run testfile.form
trace: built-in: git 'add' '--dry-run' 'testfile.form'
trace: run_command: 'n:/bin/xsltproc.exe xsl/DXLClean.xsl -'
-:2: warning: failed to load external entity "xmlschemas/domino_8_5_3.dtd"
<!DOCTYPE form SYSTEM 'xmlschemas/domino_8_5_3.dtd'>
                                                ^
add 'testfile.form'

Now we seem to be getting somewhere. Don't know about that warning, though.

camac commented 11 years ago

Ok great work, maybe adding the .exe fixed the original problem? We may soon find out! I am guessing it is some sort of problem with PATH or PATHEXT and the way that windows and git bash work. I am still getting my head around that and trying to come up with a configuration to cover all situations. I think main reason I left .exe off was so it would work it both environments.

What you say about list separators may having something to do with it? I am just taking a wild guess but maybe your PATH is built differently with different list separators? maybe git has a problem with this? When it couldn't find xsltproc exactly in that dir it may have started some type of PATH search and maybe here is where some problems where caused? maybe specifying it exactly prevented this search.

if it is not too much trouble could you show me a small anonymized sample of these commands from Git bash echo $PATH echo $PATHEXT and these commands from windows cmd echo %PATH% echo %PATHEXT% I am wondering whether the list separator is different in the windows one, and whether .exe is present in both PATHEXT

Ok on to the next problem...

How did you export the DXL for testfile.form? I think you may have exported the DXL using Tools Edit DXL or something like that am I right? My DXL files don't have the DOCTYPE definition in them. If you use Team Development -> 'Set Up source Control for this NSF'? I don't think they include the DOCTYPE definition? but I would like to confirm how it came to be in your testfile in case it is happening for others.

If I guessed right, I replicated your problem using Tools -> Edit DXL, and then pasted to a new file. I got the same error as you failed to load external entity...

What is happening is that xsltproc is trying to validate the XML against the DTD specified, and can't find the DTD. I think if you run xsltproc xsl/DXLClean.xsl testfile.form you will see the correct output, but you may also see this error message as a warning at the top? Add another argument to the xsltproc clean filter, --novalid which will skip the DTD loading phase, this worked for me so:

filter.dxlmetadata.clean=n:/bin/xsltproc.exe --novalid xsl/DXLClean.xsl -

If this works for you, I will add this into the dora setup process so it will cover the case when DXL is exported with DOCTYPE. I don't think there is a need to validate the document at this stage anyway.

Let me know how you go I hope this does the trick!

P.s. Are you on twitter? I am @gregorbyte

oysteinjaren commented 10 years ago

Hi. Sorry about the long delay. I've been pressed for time to pursue this further in the past weeks. But I will try to conclude this matter now.

echo $PATH in Git bash outputs this (I added the line breaks for clarity):

/n/bin:
.:
/usr/local/bin:
/mingw/bin:
/bin:
... (and so on)

echo %PATH% in cmd outputs this (I added the line breaks for clarity):

C:\oracle\product\11.2.0\client_1\bin;
C:\oracle\product\11.2.0\client_1;
C:\WINDOWS\system32;
C:\WINDOWS;
... (and so on)

echo $PATHEXT in Git bash outputs this:

.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

echo %PATHEXT% in cmd outputs the same:

.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

So the list separator is different for PATH. ; in cmd, : in Git Bash. For PATHEXT it is ; in both cmd and Git Bash. .EXE is present in both.

As to how I exported the DXL for testfile.form: I simply used the file found in the on-disk project. (Yes, that means I used 'Team Development' -> 'Set Up Source Control for this Application'.) All the DXL files in the on-disk project seem to have the DOCTYPE definition in them (not .metadata nor .config files).

The --novalid argument got rid of the warning about failing to load the DTD file.

Thanks for all your help. I'm going to try to apply this tool properly now and see what happens.

rPraml commented 10 years ago

We had the same problem. We solved this by adding a useless "/" to the path. So I suggest to change the path

clean = n:/bin/xsltproc.exe xsl/DXLClean.xsl -
smudge = n:/bin/xsltproc.exe xsl/DXLSmudge.xsl -

to

clean = n://bin/xsltproc.exe xsl/DXLClean.xsl -
smudge = n://bin/xsltproc.exe xsl/DXLSmudge.xsl -

or

clean = n:/bin//xsltproc.exe xsl/DXLClean.xsl -
smudge = n:/bin//xsltproc.exe xsl/DXLSmudge.xsl -

I know it makes absolutely NO sense, but it solved the problem. (I think certain path length will cause the problem)

camac commented 10 years ago

Thanks Roland!

JamesRiley commented 10 years ago

I have had the same problem and adding the additional / also worked. Very timely, thanks!!!

Would there be an easy way to fix the install script to resolve this?

sbasegmez commented 10 years ago

Same error at me, same workaround worked :)

jesse-gallagher commented 9 years ago

Roland, I can't believe that worked, but thanks!