search

Ghostkeeper / X3GWriter

X3G output plugin for Cura
GNU Affero General Public License v3.0
48 stars 16 forks source link

Mac OS X 10.14.6 Cura 4.3.0 X3GWriter 1.1.10 FlashForge Creator Pro saving zero byte x3g #30

Closed mikkyo9 closed 3 years ago

mikkyo9 commented 5 years ago

2019-10-21 22:28:35,242 - DEBUG - [Thread-7] X3GWriter.X3GWriter.write [50]: Using configured machine: fcp (FlashForge Creator Pro) 2019-10-21 22:28:35,249 - DEBUG - [Thread-7] X3GWriter.X3GWriter.gpx_executable [109]: GPX executable: /Users/mikkyo/Library/Application Support/cura/4.3/plugins/X3GWriter/X3GWriter/gpx_macos 2019-10-21 22:28:35,251 - DEBUG - [Thread-7] X3GWriter.X3GWriter.gpx_command [130]: GPX command: /Users/mikkyo/Library/Application Support/cura/4.3/plugins/X3GWriter/X3GWriter/gpxmacos -g -p -m fcp /var/folders/b/95l86vcs4xd16wqzf33ymk40000gp/T/tmptmyrr25h /var/folders/b/95l86vcs4xd16wqzf_33ymk40000gp/T/tmpwbiq8zsp 2019-10-21 22:28:35,311 - DEBUG - [Thread-7] X3GWriter.X3GWriter.write [62]: (b'', b'dyld: Library not loaded: /usr/local/lib/gcc/5/libgcc_s.1.dylib\n Referenced from: /Users/mikkyo/Library/Application Support/cura/4.3/plugins/X3GWriter/X3GWriter/gpx_macos\n Reason: image not found\n')

mikkyo9 commented 5 years ago

Looks like a libgcc dependency is required and missing

Ghostkeeper commented 5 years ago

Hmm, well since I don't have MacOS, I'm not able to compile it. I've previously asked a friend to test it on his MacOS and there it did run (but he is a developer so he'd have GCC installed fully I guess).

What I did was take the GPX binary from their releases and rename it to prevent a name collision with the Linux version.

To test, could you try these steps?

  1. Download the .dmg file for GPX from their releases.
  2. Open up the archive and extract the GPX executable to a directory.
  3. Run GPX in a terminal. Like ~/gpx --help or something.

Does it still complain about libgcc?

mikkyo9 commented 5 years ago

Sorry, haven't had time due to power outages, work, and fire chaos.

I got a chance to try things today, gpx runs fine, even the installed one. $ /Users/mikkyo/Library/Application\ Support/cura/4.3/plugins/X3GWriter/X3GWriter/gpx_macos -? GPX 2.5.2 ---snip--- its only the X3GWriter.X3GWriter.write routine that seems to be choking without the libgcc, which seems to be causing the zero byte file(maybe moving it from the temp location to the target save location?).

If I copy the gpx-macos command line and run it manually redirecting the output to my home directory, I get a valid sized file.

I tried saving a gcode file, and then converting it with the gpx-macos command as well. Will have to print it and see what comes out.

mikkyo9 commented 5 years ago

The generated x3g doesn't print.

Ghostkeeper commented 5 years ago

which seems to be causing the zero byte file

If anything in the process fails, the plug-in tends to generate a zero-byte file. This is some technicality from Cura as it creates a file first before allowing the plug-in to write to it.

Ghostkeeper commented 5 years ago

When I'm home I'll try updating the GPX version inside the package for you. Maybe there is some fix in the meanwhile. My version is 2 commits behind. Hoping you can try it for me.

I'll need to update the package anyway for the 7.0.0 SDK version increment.

Ghostkeeper commented 5 years ago

The GPX version in my plug-in is bit-for-bit identical to the latest release that I took the binary from. I'm unable to compile for MacOS myself or provide that libgcc library since I don't have an Apple computer, so there's no real way that I can find to debug or fix this for you.

There's a possible workaround here: https://github.com/haskell/haskell-platform/issues/231#issuecomment-177530716

So that would be these commands in the terminal:

brew tap homebrew/versions
brew install homebrew/versions/gcc5
mikkyo9 commented 5 years ago

That works and it lets me save the file.

mikkyo9 commented 5 years ago

I took a 20mm retraction cube stl, and ran it through the X3Gwriter, then tried to print it. The bed moved to the bottom and it tried printing in the air, so not sure if the Flash Forge Creator Pro settings are bad, or the x3g output is bad.

Ghostkeeper commented 5 years ago

I'm suspecting that the problem lies with the printer settings there. What printer are you using in Cura? Because there is no definition for FlashForge Creator Pro as far as I know. We'll probably need to configure a few settings to match the FlashForge firmware settings. X3G encodes steps per mm in there so Cura needs to know those and the position of the endstops.

mikkyo9 commented 5 years ago

I deleted and reinstalled everything. Re-set up the printer configs, installed gcc5 and have it working now. Thanks for figuring out where to get that and helping solve why it was required.

I'm using the Flash Forge for Cura profile files from here https://www.thingiverse.com/thing:3569113

I also pulled just the single needed lib from the full gcc5 install and left it in place, deleting everything else, and gpx-macos works fine now, even from the command line.

Since this libgcc_s.1.dylib is required, you could just take that single.dylib and embed it in the X3GWriter itself, and then use the install_name_tool to change the expected install path to it to be in your plugin. That would prevent folks from having to install anything else.

Ghostkeeper commented 5 years ago

The FlashForge Creator profiles you linked to already specify the profile for FlashForge Creator Pro, like this:

"machine_x3g_variant": "fcp",

in the metadata of the flashforgecreatorpro.def.json. Those profiles seem good to me. That guy did his research.

I'll look into the install_name_tool trick. But it might take a while because I'm about to go on vacation to a place where I don't have internet and also can't ask a colleague or something to help me try things on MacOS. Keeping this ticket open in the meanwhile.

eugr commented 4 years ago

Looks like the issue is not related to GPX executable at all, but to Python version that Cura is using on MacOS. So if you run the same command in the terminal it works fine with the same binary, but it doesn't work via Popen from the plugin...

Ghostkeeper commented 4 years ago

The MacOS builds of Cura are running Python 3.5.2 (like all platforms; Ultimaker builds those from the same source). I don't think it should matter though; the Popen functionality of Python hasn't changed much. Maybe it's about the working directory of the executable?

eugr commented 4 years ago

@Ghostkeeper - it looks like the dependency on gcc5 libraries are related to bundled Python, not GPX binary itself as it (gpx) runs just fine without gcc5 in the system. It only complains when running from Cura via popen.

alyssafriesen commented 4 years ago

So that would be these commands in the terminal:

brew tap homebrew/versions
brew install homebrew/versions/gcc5

This ultimately worked for me as well, although when I tried to run homebrew/versions is got hungup on trying to log in and homebrew/versions is deprecated, so I ran brew install gcc5 and I was able to save successfully

GaryL8N commented 4 years ago

brew install gcc5 did not work for me on Catalina. Kept getting:

This formula either does not compile or function as expected on macOS versions newer than High Sierra due to an upstream incompatibility

I just installed the latest gcc with brew install gcc and created a symlink to the relevant file:

mkdir /usr/local/lib/gcc/5
ln -s /usr/local/lib/gcc/9/libgcc_s.1.dylib /usr/local/lib/gcc/5/libgcc_s.1.dylib

This allowed me to create the x3g file and print.

BenBergman commented 4 years ago

It looks like gcc5 was moved to homebrew/core, so I did

brew tap homebrew/core
brew install homebrew/core/gcc5

It warned me that I should be using gcc@5 instead, but at least I'm getting x3g output files now! :)

Ghostkeeper commented 4 years ago

As BenBergman found out in #35, apparently calling the executable directly on the command line works, but calling it from Python doesn't. Based on that I'd like to ask one of you to test this for me on MacOS. I'd like to see if the working directory has anything to do with it, so could you run this on a command line?

cd "~/Library/Application Support/cura/4.6/plugins/X3GWriter"
./gpx_macos

This should give you an error that you need to provide an input file or enable standard I/O. Means that the program was running correctly. Apparently this works fine. But now try this:

cd ~
"Library/Application Support/cura/4.6/plugins/X3GWriter/gpx_macos"

This is more like how Cura is calling the application. Maybe this behaves different because the working directory is different? Just a hunch.

BenBergman commented 4 years ago

I uninstalled gcc5 to try running from the different locations (note that my gpx_macos is actually in ~/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter - double "X3GWriter"). While Cura is showing the errors again, CLI GPX shows the same output from both locations.

As an aside, with gcc5 installed I had been able to generate a non-zero x3g file. I was just able to attempt printing it but the printer did not recognize the file (even the zero byte files from pre-gcc5 were shown on the printer screen). Doing the manual gcode - CLI GPX conversion got the file to show up. I forgot to compare the x3g files to see if they are different though.

Ghostkeeper commented 4 years ago

Well as long as we know whether or not the executable gets executed is enough. I wouldn't expect the results to be any different.

I really wish now that I had a MacOS installation somewhere on which I could try stuff, but I don't. The Cura team has a Macbook but it's been out of commission the past few weeks. The question remains why calling the GPX binary from python (subprocess.Popen("/path/to/GPX")) is not working while calling GPX directly works fine.

DJPlaya commented 4 years ago

+1 Confirming this Issue on Win 10 1803 with Cura 4.6.2

Ghostkeeper commented 4 years ago

That's strange though. Previously this was specific to MacOS. I'm not seeing this problem myself on Windows. Is there something blocking GPX from executing, like a firewall? Maybe there's something in the log file? It seems to work for most people.

Ghostkeeper commented 4 years ago

Got a second report here about Windows: https://github.com/Ultimaker/Cura/issues/8093 There I can debug since I have a Windows installation. I'll give it a try in the weekend. It's been a few months since the last release when I tested on Windows, so maybe some Windows update broke it.

Ghostkeeper commented 4 years ago

On Windows I'm also still getting normal X3G files: MM_cube.zip It must be something local on those computers. It appears to be unrelated to the dylib problem on MacOS though.

DJPlaya commented 4 years ago

I cant send the Logs right now, would need 2 Weeks from now since i am on Holiday. The System is a clean Win 10 Pro Installation, ive tried it with Build 1803 and 2004 - both with the same Results. The System dosent have an Anti-Vir beside Win Defender, the Win Firewall is deactivated since the System isent connected to the Inet.

apaier commented 4 years ago

Unsure if I should create a new issue for this, or continue on this one. On Catalina (10.15.5) the gcc5 and gcc@5 won't install due to (I'm assuming) 32/64-Bit compatibility issues.

Per the request above I did - cd ~/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter ./gpx_macos

As expected this returned Command line error: provide an input file or enable standard I/O

But note- the path I had to use to get that failure has X3GWriter in it 2 times.

I then tried -

. ~/Library/Application\ Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos

(Which I think is what you wanted) This returns a different set of errors - /gpx_macos:1: command not found: \M-O\M-z\M-m\M-~^G /gpx_macos:2: command not found: 7^A /gpx_macos:3: command not found: 7^A `/gpx_macos:3: bad pattern: ^A^@^A^@^@^@:[^@^@^@^@^@^@^P``

I'm updating my gcc to the current version. I'll try the file copy approach recommended in the thread if it still applies and update the issue.

Let me know how else I can help.

apaier commented 4 years ago

I can confirm that with gcc 10 installed on Catalina, making the symbolic link as mentioned by @GaryL8N I get a non-zero byte file. Correction - this does did not work. I looked at the wrong file. For reference those commands are -

mkdir /usr/local/lib/gcc/5 ln -s /usr/local/lib/gcc/10/libgcc_s.1.dylib

This is easy to undo if you want me to be a tester for any approaches / fixes. Also willing to do a screen share for more realtime debugging if someone wants.

apaier commented 4 years ago

From the comment from @Ghostkeeper on October 22, I installed the GPX directly using brew install gpx which installed 2.5.2 with "catalina" in the formula name. Using that at the command line works.

To double check, I also downloaded the GPX release and put it in its own folder. Attempting to run there got the same error set I reported attempting to call above - /Downloads/gpx:1: command not found: \M-O\M-z\M-m\M-~^G /Downloads/gpx:2: command not found: 7^A /Downloads/gpx:2: command not found: 7^A /Downloads/gpx:3: bad pattern: ^A^@^A^@^@^@:[^@^@^@^@^@^@^P

So the version from home-brew appears to work, but the version that is just copying the binary to a location and invoking that directly fails. That failure appears to be the same as seen when attempting to invoke the gpx bundled in this plugin.

Ghostkeeper commented 4 years ago

Thanks for the help!

Is the binary from Home Brew different from the binary on GPX's Github page? I've copied the one from Github into the plug-in release. Maybe the binary from Home Brew is statically linked to libgcc. Or maybe Home Brew does some things around the installation of GPX to make it work.

I think that I can't just link libgcc myself on the first run of the plug-in though. Wouldn't you need administrator privileges to create a link like that? Its location might also depend on the MacOS version (or the GCC version).

apaier commented 4 years ago

I corrected my statement. The link approach didn't work. I'll try and track down today where home-brew has the bits for GPX and compare that to the one in the binary. I'll also see what happens if I try to use GPX from their distribution rather than home-brew to see if the behavior differs. I'll update when I've run those to ground.

apaier commented 4 years ago

Update - I uninstalled the home brew version of GPX and confirmed it was gone (the command failed). I removed the gcc hack above from the file system. I downloaded the gpx distribution copied it to my OS, and executed it from the terminal. It worked with the "provide an input file" response.

The md5 for that gpx file is MD5 (gpx) = b8d37d817e0e7ccead285833dc3f763e

This is the same as the one in the X3GWriter build - MD5 (./gpx_macos) = b8d37d817e0e7ccead285833dc3f763e

So it seems like the binaries are the same. The problem is how it is getting invoked. I can confirm this as I have the downloaded gpx file in my Downloads folder. If I'm in that folder and issue the command to either gpx binary as ./<path>/<gpx_binary_name> it works, but if I do what you say is happening . /<path>/<gpx_binary_name> It fails with that weird error. The space you said is there is the problem, but that feels like it would always have been a problem.

Attempting to troubleshoot I tailed the cura log file. In there I can still see the gcc5 reference, so it feels like something in the execution path is based off of a flawed assumption -

2020-07-31 16:22:42,394 - DEBUG - [Thread-6] X3GWriter.X3GWriter.write [50]: Using configured machine: fcp (FlashForge Creator Pro)
2020-07-31 16:22:42,395 - DEBUG - [Thread-6] X3GWriter.X3GWriter.gpx_executable [109]: GPX executable: /Users/andrewpaier/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos
2020-07-31 16:22:42,396 - DEBUG - [Thread-6] X3GWriter.X3GWriter.gpx_command [130]: GPX command: /Users/andrewpaier/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos -g -p -m fcp /var/folders/_c/p3ywh_j57p57svsf3x9xl3bc0000gn/T/tmpvrw2pk8n /var/folders/_c/p3ywh_j57p57svsf3x9xl3bc0000gn/T/tmpnqwrpvpl
2020-07-31 16:22:42,426 - DEBUG - [Thread-6] X3GWriter.X3GWriter.write [62]: (b'', b'dyld: Library not loaded: /usr/local/lib/gcc/5/libgcc_s.1.dylib\n  Referenced from: /Users/andrewpaier/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos\n  Reason: image not found\n')
2020-07-31 16:22:42,435 - DEBUG - [Thread-6] UM.FileHandler.WriteFileJob.run [80]: Writing file took 0.1769731044769287 seconds
2020-07-31 16:22:43,344 - INFO - [MainThread] UM.TaskManagement.HttpRequestManager._onRequestFinished [408]: Request [79c8c97e945545998ac97663488d45a8] finished.
2020-07-31 16:22:43,346 - DEBUG - [MainThread] UM.TaskManagement.HttpRequestManager._processNextRequestsInQueue [317]: No more requests to process, stop
2020-07-31 16:22:43,347 - INFO - [MainThread] SliceInfoPlugin.SliceInfo._onRequestFinished [275]: SliceInfo sent successfully
2020-07-31 16:22:52,584 - DEBUG - [MainThread] cura.AutoSave._onTimeout [60]: Autosaving preferences, instances and profiles

From my command prompt the call /Users/andrewpaier/Library/Application\ Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos works. As does the full command.

The error says the problem is in X3GWriter.X3GWrighter.write that looks to me like the python library and not the X3G code. I added my own debug statements to the python code and I can see it gets to the process.wait() and then throws an error. However that error isn't caught by the exception handler it actually comes through as output and is logged in the Logger.log("d",str(output)) as if everything worked.

I'm not a python guy at all, but I'll look and see if the problem is that the python invocation of the command is causing a problem. It feels like it is at this point, since the command it claims it is invoking actually works when I attempt it at the command line.

Ghostkeeper commented 4 years ago

Thanks for your help. My suspicion now is that the environment that Cura is running in is more at fault. Maybe it's not got that libgcc on its linkage path or something. I can also imagine that MacOS is protecting plug-in binaries from calling upon system binaries for security reasons.

bpatters commented 4 years ago 
2020-08-18 18:30:48,910 - DEBUG - [Thread-3] X3GWriter.X3GWriter.write [50]: Using configured machine: fcp (FlashForge Creator Pro)
2020-08-18 18:30:48,911 - DEBUG - [Thread-3] X3GWriter.X3GWriter.gpx_executable [109]: GPX executable: /Users/bpatterson/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos
2020-08-18 18:30:48,913 - DEBUG - [Thread-3] X3GWriter.X3GWriter.gpx_command [130]: GPX command: /Users/bpatterson/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos -g -p -m fcp /var/folders/53/ztlsn0nn6hv47y7q72hm3lgm0000gn/T/tmp6jh_eex7 /var/folders/53/ztlsn0nn6hv47y7q72hm3lgm0000gn/T/tmpv22ldiqv
2020-08-18 18:30:48,954 - DEBUG - [Thread-3] X3GWriter.X3GWriter.write [62]: (b'', b'dyld: Library not loaded: /usr/local/lib/gcc/5/libgcc_s.1.dylib\n  Referenced from: /Users/bpatterson/Library/Application Support/cura/4.6/plugins/X3GWriter/X3GWriter/gpx_macos\n  Reason: image not found\n')

Then using: mkdir /usr/local/lib/gcc/5 ln -s /usr/local/lib/gcc/8/libgcc_s.1.dylib

fixed it for me. Basically it's looking in /usr/local/lib/gcc/5 and it's not there.

Ghostkeeper commented 4 years ago

That's the same as what Apaier and GaryL8N said (but a different version of GCC). It's a decent workaround, but impossible for me to implement in the plug-in.

Maybe we'd need a statically linked version of the GPX executable?

jimarcher commented 4 years ago

I have a Mac running Catalina 10.15.5 with Cura doing the zero byte file thing. Can I help test? I tried creating the link @bpatters suggested 29 days ago but no joy.

Ghostkeeper commented 4 years ago

My current problem is that I don't really have the means to create a build of GPX for MacOS that statically links this libgcc library. I believe that statically linking libgcc would solve this issue since it wouldn't need to dynamically load anything then, and the correct version of the required library would get shipped along with the plug-in.

There are some inherent security issues with it though. If any of you would help me with compiling this, they are downloading a binary blob from some person on the internet which nobody can verify the integrity of. Currently people have to trust Markwal for the GPX source code which is semi-verifiable assuming the GPX executable is reproducible on Mac. And people have to trust me which is easy since the Python code is stored in the open on their computers. But if one of you provides a build of GPX with statically linked libgcc they'd have to trust one more person to not inject malicious code, and this trust is not easily verified since linking libgcc statically is harder to reproduce byte-for-byte.

I think the ideal situation is that I'd get my hands on a computer running MacOS and build the GPX executable with static linking myself. They'd still only have to trust Markwal and me, and I will be harder to trust since the GPX executable is harder to reproduce, but it's a little bit better open-source wise. But I don't have a Macintosh right now, and due to COVID I'm usually working from home, where there are no Macs.

Ghostkeeper commented 3 years ago

Markwal created a new release of GPX that apparently includes bug fixes to the way the MacOS binary is built: https://github.com/markwal/GPX/releases/tag/2.6.7 (particularly https://github.com/markwal/GPX/commit/3864548d6342c5909cba1a4fcc51f045f7134e6a). It's probably good to update GPX regardless, but updating might also solve this issue.

Ghostkeeper commented 3 years ago

I have a new release of the X3GWriter in progress. Would anyone like to try this release candidate on MacOS? X3GWriter7.0.0.zip I'll also ask some colleagues if they can test it on MacOS, but I'd like to know if this fixes the issue with 0-byte output. I hope that this fixes the issue. I don't really know how to approach it otherwise, since I don't have a MacOS computer to develop in.

To install this version, drop the .curapackage file onto the Cura window. Then restart Cura.

To uninstall it, if anything is awry or whatever, go to the Marketplace, and in the "installed" tab find the X3GWriter plug-in and press the "Uninstall" button. You could then re-install the version from the Marketplace.

Edit: For my colleague the plug-in worked on MacOS (Catalina).

Ghostkeeper commented 3 years ago

I'll assume it is fixed then.