search

mattburns / exiftool.js

A pure javascript implementation of exiftool
MIT License
64 stars 13 forks source link

can't pass image in node.js Buffer to decode #13

Closed dfries closed 9 years ago

dfries commented 9 years ago

The current node.js interface is limited to passing fs and url for exiftool.js to fs.open with. I now have two independent programs that have a node.js buffer, but not a file, one is streaming images over the network, the other is processing e-mail.

exiftool.js reads the file and puts it in an image anyway, so it was easy enough to add, and pull request #12 includes support.

dfries commented 9 years ago

Did you realize that your merge change, modified the API for the existing getExifFromLocalFileUsingNodeFs? It was returning, callback(error) or callback(null, exif_obj, url), as getExifFromNodeBuffer didn't take a url, there wasn't one to return so now it is callback(error) or callback(null, exif) no url. I wasn't using that call, so it doesn't break anything for me, I just thought I would point that out.

I see your comment on the repository history being large, I agree. Git supports editing the repository history, I've already isolated part of the history of your repository of just the exiftool.js and top directory files, which results in a 300KB repository for my own use. That way I can put it in my node_modules without without taking up much space. If you want I could remove the images out of the history, split out test and their history into a separate repository (that way you could check out that repository into test/ and everything will continue to run), and keep the history of exiftool.js and the other top level files in the current repository. I could keep all the revisions of all the files (other than the images), just some would be in one repository and some in the other.

mattburns commented 9 years ago

Sorry for the delayed response... The api for 'getExifFromLocalFileUsingNodeFs' is a bit stupid (returning the given url is weird). It was only being used by test code and actually, using getExifFromNodeBuffer makes more sense. I need to have a tidy-up!

Yeah, the huge history is a pain. I already have another repository that is intended to hold all the big assets (https://github.com/mattburns/exiftool.js-dev-dependencies). I'm not really sure how to solve my problems though. My ideal solution has the following attributes:

  1. checking out exiftool.js is small and fast
  2. contributors should be able to test any modifications easily eg: npm install; npm test or similar.
  3. I feel tests should be something you run from within the exiftool.js project. For example, asserts and expected test results should be atomic and in sync with the code. Fixes (and tests that prove the fix) are committed together and close the issues raised in this repository.
  4. running the tests should highlight if tag support has improved (and highlight regressions etc). Currently, I do this quite hackily by seeing what changes in the diff of the generated coverage report. (This is additionally hindered by issue #8)

Any ideas how I can do this?

dfries commented 9 years ago

Read up on git submodules. http://git-scm.com/book/en/v2/Git-Tools-Submodules

1, I can split up your repositories, like wrote it is tiny compared to the current repository, all the commit ids would change except the root commit, which in this case is worth it. That's a one time change. 2, you can point to the large repository on github as a submodule, git submodule init git submodule update, That will download into a subdirectory such as tests/ and pick which commit the submodule should point at. Rerun update to when the submodule changes. From there your scripts run like normal. You would need to point to the non-ssh access so it will work for most people, that will be an extra step for you to add a remote that you can use to push up with, git remote add githubssh git@github.com:mattburns/BIG_REPO.git or something, not a big deal if you know how to do it. 3, The workflow on your side is make changes, commit IN the submodule, push submodule commits to github, go to the exiftool.js repository commit your code changes like normal, git commit tests which will make a commit in exiftool.js which tells exactly what commit the tests repository needs to point at. To exiftool.js the submodule is almost another file to keep track of, but instead of a file it records what commit it should be at. Push up your exiftool.js changes to github. As long as you commit your tests submodule and your exiftool.js changes (which keeps track of what tests/ should be at), everything stays in sync. You can check out an old exiftool.js, git submodule update and it will give you the corresponding commit in the submodule. It isn't without pitfalls but I expect it will give you want you are asking for. One pitfall is forgetting to push the submodule changes up, then someone else checksout exiftool.js and submodule update fails because github doesn't have that commit. Another is submodule update will force the submodule to whatever commit is specified, so if you go to make a change you'll want to make sure you first checkout a branch or create a new branch where you are, otherwise you're in a detached head state and it's easy to loose your work if you do another submodule update, because it will force it someplace else. The pitfall problems are a user education thing, once you know what is going on, and how, it's not a problem. 4, It sounds like your test output is committed in git, so git diff will continue to work, other than that I don't see it as something git would address.

mattburns commented 9 years ago

Great stuff, thanks for your help on this. I read that article on submodules, that's all new to me. (I've been using git for a few years but depressingly on my own so never much more than add>commit>push!).

If I understood it all correctly, I think I've sorted it. I moved it all into a repo for the test stuff (no longer using node dev-dependencies mechanism). I've updated the readme with how I intend to work on the tests in future.

I still have the large history problem to sort out...

Thanks again, Matt

On 10 April 2015 at 06:03, David Fries notifications@github.com wrote:

Read up on git submodules. http://git-scm.com/book/en/v2/Git-Tools-Submodules

1, I can split up your repositories, like wrote it is tiny compared to the current repository, all the commit ids would change except the root commit, which in this case is worth it. That's a one time change. 2, you can point to the large repository on github as a submodule, git submodule init git submodule update, That will download into a subdirectory such as tests/ and pick which commit the submodule should point at. Rerun update to when the submodule changes. From there your scripts run like normal. You would need to point to the non-ssh access so it will work for most people, that will be an extra step for you to add a remote that you can use to push up with, git remote add githubssh git@github.com:mattburns/BIG_REPO.git or something, not a big deal if you know how to do it. 3, The workflow on your side is make changes, commit IN the submodule, push submodule commits to github, go to the exiftool.js repository commit your code changes like normal, git commit tests which will make a commit in exiftool.js which tells exactly what commit the tests repository needs to point at. To exiftool.js the submodule is almost another file to keep track of, but instead of a file it records what commit it should be at. Push up your exiftool.js changes to github. As long as you commit your tests submodule and your exiftool.js changes (which keeps track of what tests/ should be at), everything stays in sync. You can check out an old exiftool.js, git submodule update and it will give you the corresponding commit in the submodule. It isn't without pitfalls but I expect it will give you want you are asking for. One pitfall is forgetting to push the submodule changes up, then someone else checksout exiftool.js and submodule update fails because github doesn't have that commit. Another is submodule update will force the submodule to whatever commit is specified, so if you go to make a change you'll want to make sure you first checkout a branch or create a new branch where you are, otherwise you're in a detached head state and it's easy to loose your work if you do another submodule update, because it will force it someplace else. The pitfall problems are a user education thing, once you know what is going on, and how, it's not a problem. 4, It sounds like your test output is committed in git, so git diff will continue to work, other than that I don't see it as something git would address.

— Reply to this email directly or view it on GitHub https://github.com/mattburns/exiftool.js/issues/13#issuecomment-91430021 .

dfries commented 9 years ago

When you get into Git there's a lot there, much more than just commit/push git commit -av and you can skip the add step for existing files. I particularly like the history editing and branching options, it can really help testing and trying alternatives, and especially if multiple people are working together.

I added myself a todo item to clean up the history this weekend if you want. I just need to know if you want the large stuff to be removed and discarded, or if you want it moved into a separate repository with the history for those files intact?

Or if you want to do it yourself that's fine, I would be using gitk to look at the history and git filter-branch to make the changes. The man page is a must read for that command.

On Fri, Apr 10, 2015 at 09:45:06AM -0700, Matt Burns wrote:

Great stuff, thanks for your help on this. I read that article on submodules, that's all new to me. (I've been using git for a few years but depressingly on my own so never much more than add>commit>push!).

If I understood it all correctly, I think I've sorted it. I moved it all into a repo for the test stuff (no longer using node dev-dependencies mechanism). I've updated the readme with how I intend to work on the tests in future.

I still have the large history problem to sort out...

Thanks again, Matt

On 10 April 2015 at 06:03, David Fries notifications@github.com wrote:

Read up on git submodules. http://git-scm.com/book/en/v2/Git-Tools-Submodules

1, I can split up your repositories, like wrote it is tiny compared to the current repository, all the commit ids would change except the root commit, which in this case is worth it. That's a one time change. 2, you can point to the large repository on github as a submodule, git submodule init git submodule update, That will download into a subdirectory such as tests/ and pick which commit the submodule should point at. Rerun update to when the submodule changes. From there your scripts run like normal. You would need to point to the non-ssh access so it will work for most people, that will be an extra step for you to add a remote that you can use to push up with, git remote add githubssh git@github.com:mattburns/BIG_REPO.git or something, not a big deal if you know how to do it. 3, The workflow on your side is make changes, commit IN the submodule, push submodule commits to github, go to the exiftool.js repository commit your code changes like normal, git commit tests which will make a commit in exiftool.js which tells exactly what commit the tests repository needs to point at. To exiftool.js the submodule is almost another file to keep track of, but instead of a file it records what commit it should be at. Push up your exiftool.js changes to github. As long as you commit your tests submodule and your exiftool.js changes (which keeps track of what tests/ should be at), everything stays in sync. You can check out an old exiftool.js, git submodule update and it will give you the corresponding commit in the submodule. It isn't without pitfalls but I expect it will give you want you are asking for. One pitfall is forgetting to push the submodule changes up, then someone else checksout exiftool.js and submodule update fails because github doesn't have that commit. Another is submodule update will force the submodule to whatever commit is specified, so if you go to make a change you'll want to make sure you first checkout a branch or create a new branch where you are, otherwise you're in a detached head state and it's easy to loose your work if you do another submodule update, because it will force it someplace else. The pitfall problems are a user education thing, once you know what is going on, and how, it's not a problem. 4, It sounds like your test output is committed in git, so git diff will continue to work, other than that I don't see it as something git would address.

??? Reply to this email directly or view it on GitHub https://github.com/mattburns/exiftool.js/issues/13#issuecomment-91430021 .

Reply to this email directly or view it on GitHub: https://github.com/mattburns/exiftool.js/issues/13#issuecomment-91615221

mattburns commented 9 years ago

It's a shame to lose the history, but I don't care really. Quite happy to trash it.

I saw this proposed method on stackoverflow (http://stackoverflow.com/a/13102849/276093), do you think it would achieve the same thing?

git checkout --orphan newBranch
git add -A  # Add all files and commit them
git commit
git branch -D master  # Deletes the master branch
git branch -m master  # Rename the current branch to master