search

discoproject / disco

a Map/Reduce framework for distributed computing
http://discoproject.org
BSD 3-Clause "New" or "Revised" License
1.63k stars 241 forks source link

Update to build with OTP 18 #637

Closed ghost closed 8 years ago

ilovezfs commented 8 years ago

Can this be merged? I just noticed this patch after looking at #632. Does one supersede the other?

There's some discussion of "boneyarding" the Disco formula from Homebrew, since only Erlang 18.3 is currently supported in Homebrew core: https://github.com/Homebrew/legacy-homebrew/issues/46780#issuecomment-204744013

I've opened a PR to Homebrew (Homebrew/homebrew-core#181) applying the patch from #632 but I may switch it to the patch here. The Homebrew PR is somewhat unlikely to be accepted unless at minimum upstream has expressed an intent to merge the patch being applied, so it would help to know if you intend to merge either #632 or #637 or neither.

Also, any word on a new release tag, since we'd prefer to not patch at all and use an upstream release tag instead.

(My apologies for the comment duplication on #632 as well)

ilovezfs commented 8 years ago

For now we've opted to go with this PR over #632 : https://github.com/Homebrew/homebrew-core/commit/18f6a55ae609ea4c8159e6fd093019972143d558 https://github.com/Homebrew/homebrew-core/pull/181

Please let us know when you decide how to handle Erlang >=18.1 support so we can remove the patching. Thanks.

ghost commented 8 years ago

Can someone review this?

ghost commented 8 years ago

+1 tests passing

bob@bobsan:~/Desktop/tests/disco_fix18$ sudo make test
[sudo] password for bob: 
running install
running build
running build_py
running install_lib
creating /home/bob/.local/lib/python2.7/site-packages/disco
creating /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_dir.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_hdfs.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_http.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_raw.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_url.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_s3.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_redis.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_discodb.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_disco.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/schemes/scheme_file.py -> /home/bob/.local/lib/python2.7/site-packages/disco/schemes
copying build/lib.linux-x86_64-2.7/disco/settings.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/comm_pycurl.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/cli.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/fileutils.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/dPickle.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/test.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/ddfs.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/task.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/eventmonitor.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/util.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/comm.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/sysutil.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/func.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/error.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/compat.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/disco
copying build/lib.linux-x86_64-2.7/disco/job.py -> /home/bob/.local/lib/python2.7/site-packages/disco
creating /home/bob/.local/lib/python2.7/site-packages/disco/worker
copying build/lib.linux-x86_64-2.7/disco/worker/task_io.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker
copying build/lib.linux-x86_64-2.7/disco/worker/simple.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker
creating /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/netstring.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/external.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/func.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/worker.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
copying build/lib.linux-x86_64-2.7/disco/worker/classic/modutil.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic
creating /home/bob/.local/lib/python2.7/site-packages/disco/worker/pipeline
copying build/lib.linux-x86_64-2.7/disco/worker/pipeline/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/pipeline
copying build/lib.linux-x86_64-2.7/disco/worker/pipeline/worker.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker/pipeline
copying build/lib.linux-x86_64-2.7/disco/worker/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker
copying build/lib.linux-x86_64-2.7/disco/worker/modutil.py -> /home/bob/.local/lib/python2.7/site-packages/disco/worker
copying build/lib.linux-x86_64-2.7/disco/core.py -> /home/bob/.local/lib/python2.7/site-packages/disco
creating /home/bob/.local/lib/python2.7/site-packages/clx
copying build/lib.linux-x86_64-2.7/clx/settings.py -> /home/bob/.local/lib/python2.7/site-packages/clx
copying build/lib.linux-x86_64-2.7/clx/server.py -> /home/bob/.local/lib/python2.7/site-packages/clx
copying build/lib.linux-x86_64-2.7/clx/__init__.py -> /home/bob/.local/lib/python2.7/site-packages/clx
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_dir.py to scheme_dir.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_hdfs.py to scheme_hdfs.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_http.py to scheme_http.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_raw.py to scheme_raw.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_url.py to scheme_url.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_s3.py to scheme_s3.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_redis.py to scheme_redis.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/__init__.py to __init__.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_discodb.py to scheme_discodb.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_disco.py to scheme_disco.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/schemes/scheme_file.py to scheme_file.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/settings.py to settings.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/comm_pycurl.py to comm_pycurl.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/cli.py to cli.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/fileutils.py to fileutils.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/dPickle.py to dPickle.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/test.py to test.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/ddfs.py to ddfs.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/task.py to task.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/eventmonitor.py to eventmonitor.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/util.py to util.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/comm.py to comm.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/sysutil.py to sysutil.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/func.py to func.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/error.py to error.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/compat.py to compat.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/__init__.py to __init__.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/job.py to job.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/task_io.py to task_io.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/simple.py to simple.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/netstring.py to netstring.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/external.py to external.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/func.py to func.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/__init__.py to __init__.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/worker.py to worker.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/classic/modutil.py to modutil.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/pipeline/__init__.py to __init__.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/pipeline/worker.py to worker.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/__init__.py to __init__.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/worker/modutil.py to modutil.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/disco/core.py to core.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/clx/settings.py to settings.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/clx/server.py to server.pyc
byte-compiling /home/bob/.local/lib/python2.7/site-packages/clx/__init__.py to __init__.pyc
running install_egg_info
Writing /home/bob/.local/lib/python2.7/site-packages/disco-0.0.0.egg-info
Downloading/unpacking nose
  Downloading nose-1.3.7-py2-none-any.whl (154kB): 154kB downloaded
Installing collected packages: nose
Successfully installed nose
Cleaning up...
.......
----------------------------------------------------------------------
Ran 7 tests in 0.348s

OK
dialyzer --build_plt --output_plt .dialyzer_plt \
        --apps stdlib kernel erts compiler crypto inets syntax_tools
  Creating PLT .dialyzer_plt ...
Unknown functions:
  dbg:ctp/1
  dbg:p/2
  dbg:stop/0
  dbg:tp/2
  dbg:tpl/2
  dbg:trace_port/2
  dbg:tracer/0
  dbg:tracer/2
  hipe:compile/4
  mnesia:delete/1
  mnesia:delete_object/1
  mnesia:match_object/1
  mnesia:read/1
  mnesia:transaction/1
  mnesia:write/1
  ssl:close/1
  ssl:connect/2
  ssl:connect/3
  ssl:connect/4
  ssl:controlling_process/2
  ssl:getopts/2
  ssl:listen/2
  ssl:peername/1
  ssl:send/2
  ssl:setopts/2
  ssl:sockname/1
  ssl:ssl_accept/2
  ssl:start/0
  ssl:transport_accept/2
 done in 7m58.02s
done (passed successfully)
==> goldrush (get-deps)
==> lager (get-deps)
==> bear (get-deps)
==> meck (get-deps)
==> folsom (get-deps)
==> folsomite (get-deps)
==> plists (get-deps)
==> mochiweb (get-deps)
==> master (get-deps)
==> goldrush (compile)
==> lager (compile)
==> bear (compile)
==> meck (compile)
==> folsom (compile)
==> folsomite (compile)
==> plists (compile)
==> mochiweb (compile)
==> master (compile)
dialyzer --get_warnings -Wno_return -Wunmatched_returns -Werror_handling --plt .dialyzer_plt -r master/ebin
  Checking whether the PLT .dialyzer_plt is up-to-date... yes
  Proceeding with analysis...
Unknown functions:
  folsom_metrics:new_histogram/3
  folsom_metrics:notify/2
  lager:do_log/9
  lager:md/0
  lager:safe_format/3
  lager_config:get/2
  mochijson2:decode/1
  mochijson2:encode/1
  mochiweb_http:start/1
  mochiweb_util:path_split/1
  mochiweb_util:safe_relative_path/1
  mochiweb_util:urlsplit/1
  mochiweb_util:urlunsplit/1
  plists:foreach/2
Unknown types:
  disco_util:timestamp/0
 done in 0m51.01s
done (passed successfully)
WARN:  Expected /home/bob/Desktop/tests/disco_fix18/master/deps/triq to be an app dir (containing ebin/*.app), but no .app found.
==> mochiweb (get-deps)
==> master (get-deps)
WARN:  Expected /home/bob/Desktop/tests/disco_fix18/master/deps/triq to be an app dir (containing ebin/*.app), but no .app found.
Pulling triq from {git,"https://github.com/krestenkrab/triq.git","master"}
Cloning into 'triq'...
==> triq (get-deps)
==> mochiweb (compile)
==> triq (compile)
Compiled src/triq_autoexport.erl
Compiled src/triq_expr.erl
Compiled src/triq_fsm_stub.erl
Compiled src/triq_statem_fsm.erl
Compiled src/triq_sup.erl
Compiled src/triq_statem.erl
Compiled src/triq_app.erl
Compiled src/triq_fsm.erl
Compiled src/triq.erl
Compiled src/triq_dom.erl
==> master (compile)
==> master (eunit)
Compiled test/json_validator_test.erl
Compiled test/jc_utils_test.erl
Compiled test/pipeline_utils_test.erl
Compiled test/worker_protocol_test.erl
Compiled test/ddfs_util_test.erl
test/ddfs_tag_test.erl:13: Warning: function tokentype/0 is unused
Compiled test/ddfs_tag_test.erl
Compiled src/ddfs/ddfs_web.erl
Compiled src/jobpack.erl
Compiled src/ddfs/ddfs_node.erl
Compiled src/ddfs/http_queue.erl
Compiled src/ddfs/ddfs_rebalance.erl
Compiled src/ddfs/ddfs_http.erl
Compiled src/ddfs/ddfs_put.erl
Compiled src/ddfs/ddfs_gc.erl
Compiled src/ddfs/ddfs_gc_node.erl
Compiled src/ddfs/ddfs_util.erl
Compiled src/ddfs/ddfs_master.erl
Compiled src/ddfs/ddfs.erl
Compiled src/ddfs/ddfs_gc_main.erl
Compiled src/ddfs/ddfs_get.erl
Compiled src/ddfs/ddfs_tag.erl
Compiled src/ddfs/ddfs_tag_util.erl
Compiled src/disco_proxy.erl
Compiled src/worker_inputs.erl
Compiled src/disco_util.erl
Compiled src/disco_web.erl
Compiled src/fair_scheduler.erl
Compiled src/fair_scheduler_fifo_policy.erl
Compiled src/disco_profile.erl
Compiled src/lock_server.erl
Compiled src/disco_config.erl
Compiled src/job_event.erl
Compiled src/fair_scheduler_fair_policy.erl
Compiled src/worker_utils.erl
Compiled src/worker_runtime.erl
Compiled src/work_queue.erl
Compiled src/temp_gc.erl
Compiled src/worker_throttle.erl
Compiled src/http_client.erl
Compiled src/disco_main.erl
Compiled src/worker_protocol.erl
Compiled src/fair_scheduler_job.erl
Compiled src/json_validator.erl
Compiled src/disco_server.erl
Compiled src/jc_utils.erl
Compiled src/disco_worker.erl
Compiled src/pipeline_utils.erl
Compiled src/disco.erl
Compiled src/hdfs.erl
Compiled src/job_coordinator.erl
Compiled src/web_server.erl
Compiled src/node_mon.erl
Compiled src/event_server.erl
Compiled src/discozip.erl
  All 30 tests passed.
Cover analysis: /home/bob/Desktop/tests/disco_fix18/master/.eunit/index.html
/home/bob/Desktop/tests/disco_fix18

Tested with:

bob@bobsan:~/Desktop/tests/disco_fix18$ python -V
Python 2.7.10
bob@bobsan:~/Desktop/tests/disco_fix18$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 15.10
Release:    15.10
Codename:   wily
bob@bobsan:~/Desktop/tests/disco_fix18$ cat /usr/lib/erlang/releases/18/OTP_VERSION 
18.0
pooya commented 8 years ago

Thanks. I ran some tests and this seems to be working fine.

ilovezfs commented 8 years ago

@pooya Cool, thank you. Any chance of getting a new stable tag so we can upgrade the Homebrew formula without any patching?