Possibility to publish as a Cocoapods package?

j0j00 commented 5 years ago

I'm currently in the middle of creating a Flutter package for LiquidCore, but I'm having trouble getting the iOS version to work as I don't have much experience with iOS or Xcode.

Following the instructions here and running carthage build --no-skip-current gives me the following error log.

Flutter mainly works with cocoapods, and from my experience so far, pods seem a lot easier to work with than carthage.

I made some progress writing a local LiquidCore.podspec file, but it takes more than 20 minutes to compile (I'm not sure if this is because it's running in a VM, or the source_files wildcards are too permissive) - I'm probably also missing a load of compiler flags which I'm clueless about:

#
#  Be sure to run `pod spec lint LiquidCore.podspec' to ensure this is a
#  valid spec and to remove all comments including this before submitting the spec.
#
#  To learn more about Podspec attributes see http://docs.cocoapods.org/specification.html
#  To see working Podspecs in the CocoaPods repo see https://github.com/CocoaPods/Specs/
#

Pod::Spec.new do |s|
  s.name         = "LiquidCore"
  s.version      = "0.5.1"
  s.summary      = "Provides Node.js virtual machines to run inside iOS apps."
  s.description  = <<-DESC
LiquidCore enables Node.js virtual machines to run inside Android and iOS apps. It provides a complete runtime environment, including a virtual file system and native SQLite3 support.
                   DESC

  s.homepage     = "https://github.com/LiquidPlayer/LiquidCore"
  s.license      = { :type => "MIT", :file => "LICENSE.md" }

  s.author       = { "LiquidPlayer" => "eric@flicket.tv" }

  s.platform     = :ios

  s.source       = { :git => "https://github.com/LiquidPlayer/LiquidCore.git", :tag => "#{s.version}" }

  s.source_files  = "deps/nan-2.5.1/**/*",
                    "deps/node-8.9.3/**/*", 
                    "deps/node-sqlite3/**/*", 
                    "deps/openssl-1.0.2o/**/*", 
                    "LiquidCoreCommon/**/*", 
                    "LiquidCoreiOS/LiquidCore/LiquidCore/*.h",
                    "LiquidCoreiOS/LiquidCore/API/*.{h,m,cpp}"
  # I don't think public_header_files is needed if all the headers are included in the source_files?
  # s.public_header_files = "LiquidCore/LiquidCore/*.h"

  # ――― Project Settings ――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
  #
  #  If your library depends on compiler flags you can set them in the xcconfig hash
  #  where they will only apply to your library. If you depend on other Podspecs
  #  you can include multiple dependencies to ensure it works.

  s.requires_arc = true
  s.frameworks = "JavaScriptCore"

  # s.xcconfig = { "HEADER_SEARCH_PATHS" => "$(SDKROOT)/usr/include/libxml2" }
  # s.dependency "JSONKit", "~> 1.4"

end

You don't have to run pod spec lint LiquidCore.podspec as that takes ages to run - it clones the repo from scratch on every run. I was able to workaround it by cloning the repo locally, placing the LiquidCore.podspec file in the root, and referencing it from a different Podfile in a different project.

Was there an advantage to using Carthage over Cocoapods?
Also on a side-note, did you know about the Nodekit project? I think it's abandoned now, but it did look pretty interesting.

ericwlange commented 5 years ago

I am also not really an iOS developer, though I pretend sometimes.

I got LiquidCore to build using the IDE (XCode) first, so all of the tricky build steps were handled there. When it came time to figure out how to distribute this thing, I first looked into Cocoapods, since it is the de-facto standard. I quickly lost interest because it required me to build a whole new makefile of sorts using a process and format that I've never seen before. Carthage, on the other hand, did not require me to do anything different and just (sort of) worked out of the box. It takes the xcodeproj build file and uses that without me having to create a separate makefile (in this case, podspec). What bothers me about Cocoapods is that you have to keep two build systems in sync every time you change something, which seemed more error prone than necessary.

Of course, Carthage is not a perfect answer. When I built ReactNativeSurface, I ran into all sorts of problems with Carthage that are still not completely resolved.

Version 0.5.1 is literally only the second version of iOS that I released, so it is very early days. Carthage was intended to be the quickest/dirtiest way to get it distributed, but I always intended to revisit Cocoapods. I'm glad that you've created something, because it gives me somewhere to start!

The main issues with building a podspec are that I have several script-based build phases:

Generate a node_javascript.cc file from all of the node .js files:

cd ${SRCROOT}/../deps/node-8.9.3
cp ${SRCROOT}/LiquidCore/node-8.9.3/node/config.gypi .
SCRIPT_INPUT_FILES=''
SCRIPT_INPUT_FILE_COUNT=
SCRIPT_INPUT_FILE_LIST_COUNT=
for INFILE in ${!SCRIPT_INPUT_FILE_*};
do
INF=${!INFILE}
FILTERED=`echo $INF | sed 's/.*node-8.9.3\///'`
SCRIPT_INPUT_FILES="$SCRIPT_INPUT_FILES ${FILTERED}"
done
exec tools/js2c.py ${SCRIPT_OUTPUT_FILE_0} ${SCRIPT_INPUT_FILES}

Generate a node_provider.h file from a .d file:

cd ${SRCROOT}/../deps/node-8.9.3
cp ${SRCROOT}/LiquidCore/node-8.9.3/node/config.gypi .
dtrace -h -xnolibs -s ${SCRIPT_INPUT_FILE_0} -o ${SCRIPT_OUTPUT_FILE_0}

Generate some c files from my javascript polyfills:

cd ${SRCROOT}/LiquidCore/node-8.9.3/V82JSC/polyfill
unset SCRIPT_INPUT_FILE_COUNT
printf "/* Autogenerated.  Do not edit. */\n" > ${SCRIPT_OUTPUT_FILE_0}
FILTER="$SRCROOT/LiquidCore/node-8.9.3/V82JSC/polyfill/"
for INFILE in ${!SCRIPT_INPUT_FILE_*};
do
INF=${!INFILE}
FILTERED=${INF#$FILTER}
xxd -i ${FILTERED} >> ${SCRIPT_OUTPUT_FILE_0}
printf "\n/*-------*/\n" >> ${SCRIPT_OUTPUT_FILE_0}
done

(Incidentally, it appears to be this build phase that is failing in your build - in your log file the only error is xxd: 0: No such file or directory. That either means you don't have the xxd tool installed -- easy enough for you to check -- or that it can't find some of the input files. Can you check if you have xxd and install it if not?)

I couldn't immediately figure out how to replicate these build steps in a podspec, so I demurred and moved to Carthage instead. I assume it is possible, I just didn't do it yet.

I have not seen NodeKit, no. It does indeed look interesting. I never really looked for other projects to solve this problem because of the strange history of this project:

LiquidCore grew somewhat organically. It was not originally designed to do what it does today (Node for mobile). It came out of a startup I did in 2012 which was focused on aggregating video content from a diverse set of sources, indexing the content and creating a consistent playback experience on mobile devices (the genesis of LiquidPlayer). I borrowed a number of the algorithms for extracting content from projects like youtube-dl, Kodi (then XBMC) and others. The algos were unfortunately written in Python and not suitable for mobile. Plus because of the changing nature of the sites they extract from, the algos needed to be updated frequently. Given the complexity of re-releasing an app in the app stores every time something changed, I needed a scripting language that could run locally and be downloaded from a server. Javascript was the obvious choice. I was working with iOS first at the time, and JavaScriptCore was introduced on iOS 7. So I built the entire experience there. When it came time to port to Android, I discovered that no such Javascript interface existed. So I built my own by using a GTK port of JavaScriptCore. Once the startup failed (as they are wont to do), I open sourced a cleaned up version of my Javascript project, AndroidJSCore. But then the GTK JSC engine was getting really old (and I had a ton of concurrency bugs), so I decided to use V8 instead. Node had a nice version of V8 integrated and a lot of support, so I decided to use all of node, and not just V8. It was originally called NodeDroid (you can still find references to that in the code), but then renamed LiquidCore as I figured there was no reason to limit it to Android. Everything just snowballed from there and was done in a vacuum to everything else out there.

j0j00 commented 5 years ago

Thanks for the insight into the project's history; startups are always risky, but I guess it was a good thing as it led to the creation of this library.

I can't believe I didn't notice the xxd error before (I'm not sure how you did in that massive log), I do have xxd installed, I think it just couldn't find the input files.

cocoapods' script_phases seem like the way to go. I think it's worth moving all the build path scripts and their input files into individual bash scripts, this way they can be used by both the podspec and Xcode.

I'll try doing some more sandboxing in the near future and see if I can get the podspec to work.

ericwlange commented 5 years ago

I think I know what is happening with xxd. In the script above, add the following line:

unset SCRIPT_INPUT_FILE_LIST_COUNT

Complete script:

cd ${SRCROOT}/LiquidCore/node-8.9.3/V82JSC/polyfill
unset SCRIPT_INPUT_FILE_COUNT
unset SCRIPT_INPUT_FILE_LIST_COUNT
printf "/* Autogenerated.  Do not edit. */\n" > ${SCRIPT_OUTPUT_FILE_0}
FILTER="$SRCROOT/LiquidCore/node-8.9.3/V82JSC/polyfill/"
for INFILE in ${!SCRIPT_INPUT_FILE_*};
do
   INF=${!INFILE}
   FILTERED=${INF#$FILTER}
   xxd -i ${FILTERED} >> ${SCRIPT_OUTPUT_FILE_0}
   printf "\n/*-------*/\n" >> ${SCRIPT_OUTPUT_FILE_0}
done

I think XCode added SCRIPT_INPUT_FILE_LIST_COUNT (which I can see I already fixed in the first script) which is matching the pattern for INFILE in ${!SCRIPT_INPUT_FILE_*};. XCode is interpreting it as a filename, but it is a value of zero. Hence the error message. It is looking for a file named "0".

Try that and see if it fixes the build problem. For some reason, it is not failing for me, but who knows. If it fixes the problem, I will commit the change. What I really want is a regex match that replaces the * wildcard with [0-9]* to avoid this problem in the future, but I am not sure how smart bash is. I will test it.

ericwlange commented 5 years ago

Ok, this should do it:

For the first Run Script:

cd ${SRCROOT}/../deps/node-8.9.3
cp ${SRCROOT}/LiquidCore/node-8.9.3/node/config.gypi .
SCRIPT_INPUT_FILES=''
for INFILE in ${!SCRIPT_INPUT_FILE_*};
do
    INF=${!INFILE}
    if [ -e "$INF" ]; then
        FILTERED=`echo $INF | sed 's/.*node-8.9.3\///'`
        SCRIPT_INPUT_FILES="$SCRIPT_INPUT_FILES ${FILTERED}"
    fi
done
exec tools/js2c.py ${SCRIPT_OUTPUT_FILE_0} ${SCRIPT_INPUT_FILES}

And for the third:

cd ${SRCROOT}/LiquidCore/node-8.9.3/V82JSC/polyfill
printf "/* Autogenerated.  Do not edit. */\n" > ${SCRIPT_OUTPUT_FILE_0}
FILTER="$SRCROOT/LiquidCore/node-8.9.3/V82JSC/polyfill/"
for INFILE in ${!SCRIPT_INPUT_FILE_*};
do
    INF=${!INFILE}
    if [ -e "$INF" ]; then
        FILTERED=${INF#$FILTER}
        xxd -i ${FILTERED} >> ${SCRIPT_OUTPUT_FILE_0}
        printf "\n/*-------*/\n" >> ${SCRIPT_OUTPUT_FILE_0}
    fi
done

Regex wildcards are not supported, but simply checking for the presence of the file should solve this. Working on my system.

j0j00 commented 5 years ago

The xxd line appeared to have been a red-herring, as the actual error seems to be with Xcode being unable to parse my FunctionTemplate.cpp.

I was working off the master branch, which had accidentally changed #include "V82JSC.h" to include "V82JSC.h" in the Updated license to MIT commit.

In hindsight, this was obviously an error, but since I wasn't very familiar with C++, I assumed it must've been valid and I was missing some kind of dynamic compiler option. Switching to the 0.5.1 branch seems to build Carthage just fine.

I think that's the hard part done, I'll try creating a podspec that compiles locally and then make a PR for it.

Also, is 11.2 the absolute minimum supported deployment target or could it technically work with lower iOS versions?

ericwlange commented 5 years ago

Oops! Nice catch. Cut and paste error when updating headers. Fixed in master now.

As for iOS versions, there is no particular reason 11.2 was chosen, other than that was the default for XCode when I started the project. Feel free to dial it back and see if it works on older versions. The only thing I would be concerned about is what version of JavaScriptCore exists on older versions of iOS. Node 8.9.3 (and V82JSC especially) assume certain capabilities, like most of the ES6 stuff. I make heavy use of Proxy and Reflect. So long as those are supported, then I don't anticipate a problem.

j0j00 commented 5 years ago

Just an FYI: I was able to get the podspec working a while back, I'll make a PR for it in the next couple days.

I guess there's no harm in keeping it iOS 9 upwards, and if other developers run into weird bugs, they can either fix it in a PR or bump up their minimum requirements.

ericwlange commented 5 years ago

v0.6.0 is now released and the pod has been pushed to both the main spec repo and a LiquidCore-scoped repo (https://github.com/LiquidPlayer/Specs.git). Thanks for all your help in getting this up and running. It was a huge time saver!

LiquidPlayer / LiquidCore

Possibility to publish as a Cocoapods package? #70