Slow start of Julia kernel.

[BoPeng]

Loading the three perhaps rarely used modules take 4 seconds whenever the Julia kernel is started. There should be some way for us to use them wherever they are actually used.

$ time julia -e 'using Feather;using DataFrames;using NamedArrays;'
WARNING: Method definition ==(Base.Nullable{S}, Base.Nullable{T}) in module Base at nullable.jl:238 overwritten in module NullableArrays at /Users/bpeng1/.julia/v0.6/NullableArrays/src/operators.jl:99.

real    0m4.514s
user    0m4.373s
sys 0m0.295s

$ time julia -e ''

real    0m0.327s
user    0m0.217s
sys 0m0.143s

[BoPeng]

There is a walk around to use eval to resolve this.

[HenryLeongStat]

if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))

Works.

However, the followings:

function __julia_py_repr_matrix(obj)
  if !isdefined(:Feather)
    eval(Expr(:using,:Feather))
  end
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  end
  tf = joinpath(tempname())
  p = convert(DataFrame, obj)
  Feather.write(tf, p)
  return "numpy.asmatrix(read_dataframe(r'" * tf * "'))"
end
a=diagm(1:6)
__julia_py_repr_matrix(a)

Error would appear. Run __julia_py_repr_matrix(a) again after the error, it works. Trying to figure out the problem here.

[BoPeng]

Not sure but that post appears to mention the use of DataFrames.DataFrame or something of that nature.

[HenryLeongStat]

Trying something like the followings:

function mat2df()
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  end
  if !isdefined(:DataFrames)
    print("Pakcage wasn't loaded \n \n")
  elseif isdefined(:DataFrames)
    print("Pakcage was loaded \n \n")
    try
      print("Since the package is loaded, we can create a dataframe \n \n")
      print(DataFrame(A = 1:4, B = 4.0:-1.0:1.0))
    catch
      print("package is loaded, why can't a dataframe be defined? \n \n")
      print(DataFrames.DataFrame(A = [1, 2], B = [4, 5]))
    end
  end
end

mat2df()

It does load the package, but it will raise an error:

manchongleong@Mande-MacBook-Pro:~$ julia testJ1.ji
Pakcage was loaded

Since the package is loaded, we can create a dataframe

package is loaded, why can't a dataframe be defined?

ERROR: LoadError: MethodError: no method matching DataFrames.DataFrame(; A=[1, 2], B=[4, 5])
Closest candidates are:
  DataFrames.DataFrame(; kwargs...) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:98
  DataFrames.DataFrame(!Matched::Array{DataType,1}, !Matched::Array{Symbol,1}, !Matched::Array{Bool,1}, !Matched::Integer) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:132 got unsupported keyword arguments "A", "B"
  DataFrames.DataFrame(!Matched::Array{D<:Associative,1}) where D<:Associative at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:157 got unsupported keyword arguments "A", "B"
  ...
Stacktrace:
 [1] mat2df() at /Users/manchongleong/testJ1.ji:26
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] include(::String) at ./sysimg.jl:14
 [4] process_options(::Base.JLOptions) at ./client.jl:305
 [5] _start() at ./client.jl:371
while loading /Users/manchongleong/testJ1.ji, in expression starting on line 34

Seems like the problem is the world age. In my concepts, when the function runs, it is in the "old" age, and the package is loaded, it is in the "new" age, but when the syntax based on the package runs, it is still in the "old" age. That is just my guess after I read this discussion..

[HenryLeongStat]

Btw, it does load the package, just fails to call it...

[HenryLeongStat]

Still fail after adding Base.invokelatest:

function mat2df()
  #check_defined_eval(:Feather)
  #check_defined_eval(:DataFrames)
  #if !isdefined(:Feather)
  #  eval(Expr(:using,:Feather))
  #  #backend()
  #end
  if !isdefined(:DataFrames)
    eval(Expr(:using,:DataFrames))
  #  #backend()
  end
  #tf = joinpath(tempname())
  if !isdefined(:DataFrames)
    print("Pakcage wasn't loaded \n \n")
  #  #backend()
  elseif isdefined(:DataFrames)
    print("Pakcage was loaded \n \n")
    try
      print("Since the package is loaded, we can create a dataframe \n \n")
      #return convert(DataFrames.DataFrame, rand(2,2))
      print(Base.invokelatest(DataFrame(A = 1:4, B = 4.0:-1.0:1.0)))
    catch
      print("package is loaded, why can't a dataframe be defined? \n \n")
      #eval(Expr(:import,:DataFrames))
      #error(return convert(DataFrames.DataFrame, rand(2,2)))
      print(DataFrames.DataFrame(A = [1, 2], B = [4, 5]))
    end
  end
  #return convert(DataFrames.DataFrame, obj)
  #Feather.Feather.write(tf, p)
  #return "numpy.asmatrix(read_dataframe(r'" * tf * "'))"
end

mat2df()

ERROR: LoadError: MethodError: no method matching DataFrames.DataFrame(; A=[1, 2], B=[4, 5])
Closest candidates are:
  DataFrames.DataFrame(; kwargs...) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:98
  DataFrames.DataFrame(!Matched::Array{DataType,1}, !Matched::Array{Symbol,1}, !Matched::Array{Bool,1}, !Matched::Integer) at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:132 got unsupported keyword arguments "A", "B"
  DataFrames.DataFrame(!Matched::Array{D<:Associative,1}) where D<:Associative at /Users/manchongleong/.julia/v0.6/DataFrames/src/dataframe/dataframe.jl:157 got unsupported keyword arguments "A", "B"
  ...
Stacktrace:
 [1] mat2df() at /Users/manchongleong/testJ1.ji:26
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] include(::String) at ./sysimg.jl:14
 [4] process_options(::Base.JLOptions) at ./client.jl:305
 [5] _start() at ./client.jl:371
while loading /Users/manchongleong/testJ1.ji, in expression starting on line 34

[HenryLeongStat]

I think the method mentioned in this discussion might work.

It says that we can wrap the function and the package in a new package. For example,

module __Read_DF
  export __read_DF
  using DataFrames
  function __read_DF(...)
        return ...
  end 
end
using __Read_DF

[BoPeng]

The invokelatest problem is fixed here but perhaps not in Julia yet.

[HenryLeongStat]

I see, they haven't released the new version?

[BoPeng]

I do not understand the new proposal. Would not DataFrames be loaded when __Read_DF is loaded?

[HenryLeongStat]

Seems not working, because when the module is defined, the syntaxes inside would be executed. I thought it would be like a function, and the syntaxes wouldn't be executed when it defines.

[BoPeng]

I just marked this ticket as help wanted. Let us ignore it and see if some Julia expert can help.

[HenryLeongStat]

OK. I think a lot of julia users have the same problem as well. 😄

[HenryLeongStat]

Now need to fix Python to Julia

[BoPeng]

What python to Julia? You can close this ticket if the specific issue is resolved.

[HenryLeongStat]

Test fails:

    def testGetPythonMatrixFromJulia(self):
        # Python -> Julia
        with sos_kernel() as kc:
            iopub = kc.iopub_channel
            # create a matrix 
            execute(kc=kc, code='''
import numpy as np
mat_var = np.matrix([[1,2],[3,4]])
''')
            clear_channels(iopub)
            execute(kc=kc, code="%use Julia")
            wait_for_idle(kc)
            execute(kc=kc, code="%get mat_var")
            wait_for_idle(kc)
            execute(kc=kc, code="size(mat_var)")
            res11 = get_display_data(iopub)
            self.assertEqual(res11, '(2, 2)')
            execute(kc=kc, code="%use sos")
            wait_for_idle(kc)

======================================================================
FAIL: testGetPythonMatrixFromJulia (__main__.TestJuliaKernel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_Julia_kernel1.py", line 58, in testGetPythonMatrixFromJulia
    ''')
  File "/Users/manchongleong/anaconda/lib/python3.6/site-packages/ipykernel/tests/utils.py", line 69, in execute
    validate_message(reply, 'execute_reply', msg_id)
  File "/Users/manchongleong/anaconda/lib/python3.6/site-packages/ipykernel/tests/test_message_spec.py", line 262, in validate_message
    nt.assert_equal(msg['parent_header']['msg_id'], parent)
AssertionError: 'f9d9614b-fde7fb337606b970be9a776b' != 'e2d25b4d-88c7fd586a61ea1a836fd754'
- f9d9614b-fde7fb337606b970be9a776b
+ e2d25b4d-88c7fd586a61ea1a836fd754

Does it mean it takes too long to execute the codes?

[BoPeng]

Does everything works interactively? If so, you can commit and I will have a look at the test. The test does fail from time to time due to differences in environment and it can be hard to make it work correctly.

[HenryLeongStat]

Yes, it works fine interactively. I have already pushed it. Trying to fix the test.

[HenryLeongStat]

Still pretty slow... Especially for running the testing script test/test_Julia_kernel.py