search

ROCm / aomp

AOMP is an open source Clang/LLVM based compiler with added support for the OpenMP® API on Radeon™ GPUs. Use this repository for releases, issues, documentation, packaging, and examples.
https://github.com/ROCm/aomp
Apache License 2.0
206 stars 46 forks source link

ICE: Null register has no regunits #259

Open reguly opened 2 years ago

reguly commented 2 years ago

I compiled the latest source version, and getting the following error when compiling for an MI50. Source attached below.

clang --version 
AOMP_STANDALONE_14.0-0 clang version 14.0.0 (https://github.com/radeonopencompute/llvm-project 98c673b1c90d118c7046b093e9907a0e5e9edcd4)

aompcc -g -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math omptest.c -DLOMP -DL1D -O3

llc: /home/istvan/git/aomp14.0/llvm-project/llvm/include/llvm/MC/MCRegisterInfo.h:684: llvm::MCRegUnitIterator::MCRegUnitIterator(llvm::MCRegister, const llvm::MCRegisterInfo*): Assertion `Reg && "Null register has no regunits"' failed.
PLEASE submit a bug report to https://github.com/ROCm-Developer-Tools/aomp and include the crash backtrace.
Stack dump:
0.  Program arguments: /home/istvan/rocm/aomp_14.0-0/bin/llc /tmp/omptest-e55c8f-gfx906-optimized-7c26dc.bc -O3 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 -filetype=obj --amdhsa-code-object-version=4 -o /tmp/omptest-e55c8f-gfx906-9b2e8a.o
1.  Running pass 'CallGraph Pass Manager' on module '/tmp/omptest-e55c8f-gfx906-optimized-7c26dc.bc'.
2.  Running pass 'SI post-RA bundler' on function '@__omp_outlined__.5_wrapper'
 #0 0x0000558dbaa83db4 PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x0000558dbaa814de SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f54560153c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x153c0)
 #3 0x00007f5455ab418b raise /build/glibc-eX1tMB/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #4 0x00007f5455a93859 abort /build/glibc-eX1tMB/glibc-2.31/stdlib/abort.c:81:7
 #5 0x00007f5455a93729 get_sysdep_segment_value /build/glibc-eX1tMB/glibc-2.31/intl/loadmsgcat.c:509:8
 #6 0x00007f5455a93729 _nl_load_domain /build/glibc-eX1tMB/glibc-2.31/intl/loadmsgcat.c:970:34
 #7 0x00007f5455aa4f36 (/lib/x86_64-linux-gnu/libc.so.6+0x36f36)
 #8 0x0000558db92dbc7c (anonymous namespace)::SIPostRABundler::collectUsedRegUnits(llvm::MachineInstr const&, llvm::BitVector&) const (.isra.0) SIPostRABundler.cpp:0:0
 #9 0x0000558db92dcc79 (anonymous namespace)::SIPostRABundler::runOnMachineFunction(llvm::MachineFunction&) SIPostRABundler.cpp:0:0
#10 0x0000558db9d04d98 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/home/istvan/rocm/aomp_14.0-0/bin/llc+0x1288d98)
#11 0x0000558dba1ea8b5 llvm::FPPassManager::runOnFunction(llvm::Function&) (/home/istvan/rocm/aomp_14.0-0/bin/llc+0x176e8b5)
#12 0x0000558db989ed17 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#13 0x0000558dba1ebbe2 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/home/istvan/rocm/aomp_14.0-0/bin/llc+0x176fbe2)
#14 0x0000558db90d04eb compileModule(char**, llvm::LLVMContext&) llc.cpp:0:0
#15 0x0000558db9059822 main (/home/istvan/rocm/aomp_14.0-0/bin/llc+0x5dd822)
#16 0x00007f5455a950b3 __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:342:3
#17 0x0000558db90c7f5e _start (/home/istvan/rocm/aomp_14.0-0/bin/llc+0x64bf5e)
clang-14: error: unable to execute command: Aborted (core dumped)
clang-14: error: amdgcn-link command failed due to signal (use -v to see invocation)
AOMP_STANDALONE_14.0-0 clang version 14.0.0 (https://github.com/radeonopencompute/llvm-project 98c673b1c90d118c7046b093e9907a0e5e9edcd4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/istvan/rocm/aomp_14.0-0/bin
clang-14: note: diagnostic msg: Error generating preprocessed source(s).
ERROR:  The following command failed with return code 254.
        /home/istvan/rocm/aomp_14.0-0/bin/clang  -g  -O3 -target x86_64-pc-linux-gnu -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906  -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -DLOMP -DL1D  omptest.c -o /home/istvan/a.out

omptest.txt

reguly commented 2 years ago

The culprit is -g, compiles OK without it.

However, if I swap -DL1D for -DL3D thereby changing to 3D indexed arrays, it then gives me another ICE:

clang-14: /home/istvan/git/aomp14.0/llvm-project/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp:263: void {anonymous}::CheckVarsEscapingDeclContext::markAsEscaped(const clang::ValueDecl*): Assertion `!VD->getType()->isVariablyModifiedType() && "Parameter captured by value with variably modified type"' failed.
carlobertolli commented 2 years ago

@reguly all problems you mention disappear with the latest version of the compiler. I suggest you try the latest: https://github.com/ROCm-Developer-Tools/aomp/releases

These are the flags I used for building it:

-DLOMP -DL1D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -g -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a

-DLOMP -DL3D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -g -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a

-DLOMP -DL1D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a

-DLOMP -DL3D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a

As you can see, my target is gfx90a, while I believe you were building for gfx906. That should not be the source of the issue you saw.

Running goes to completion without error for all cases. I did not verify results, but I see there is validation code in the test itself.

Please verify and close if appropriate.

reguly commented 2 years ago

@carlobertolli I tried this with Centos 7 and Ubuntu 20, but the 3D indexing ICE still persists for me. I did the userspace install into $HOME from the rpm/deb, set the environment variables. aompcc omptest.c -DLOMP -DL3D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -g -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906

Gives the following:

clang-14: /home/release/git/aomp14.0/llvm-project/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp:261: void {anonymous}::CheckVarsEscapingDeclContext::markAsEscaped(const clang::ValueDecl*): Assertion `!VD->getType()->isVariablyModifiedType() && "Parameter captured by value with variably modified type"' failed.
PLEASE submit a bug report to https://github.com/ROCm-Developer-Tools/aomp and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.  Program arguments: /home/istvan/usr/lib/aomp_14.0-1/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-pc-linux-gnu -emit-llvm-bc -emit-llvm-uselists -disable-free -clear-ast-before-backend -main-file-name omptest2.c -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=none -menable-no-infs -menable-no-nans -menable-unsafe-fp-math -fno-signed-zeros -mreassociate -freciprocal-math -ffp-contract=fast -fno-rounding-math -ffast-math -ffinite-math-only -mconstructor-aliases -target-cpu gfx906 -fcuda-is-device -fcuda-allow-variadic-functions -target-cpu gfx906 -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -mllvm -amdgpu-spill-cfi-saved-regs -gheterogeneous-dwarf -resource-dir /home/istvan/usr/lib/aomp_14.0-1/lib/clang/14.0.0 -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/lib/clang/14.0.0/include/openmp_wrappers -include __clang_openmp_device_functions.h -I /home/istvan/usr/lib/aomp_14.0-1/bin/../include -D LOMP -D L3D -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/bin/../include -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/bin/../../include -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/bin/../include -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/lib/clang/14.0.0/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/lib/clang/14.0.0/include/cuda_wrappers -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/bin/../include -internal-isystem /home/istvan/usr/lib/aomp_14.0-1/lib/clang/14.0.0/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -Wall -Wno-unused-result -Wno-unused-variable -fdebug-compilation-dir=/home/istvan -ferror-limit 19 -fopenmp -fopenmp-target-new-runtime -fgnuc-version=4.2.1 -fcolor-diagnostics -vectorize-loops -vectorize-slp -mllvm -disable-promote-alloca-to-vector=true -fopenmp-is-device -fopenmp-host-ir-file-path /tmp/omptest2-3097f4.bc -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/omptest2-c58599.gfx906.bc -x c omptest2.c
1.  /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h:7:1: current parser token 'struct'
2.  omptest2.c:33:5: LLVM IR generation of declaration 'main'
 #0 0x0000556c090a5eb4 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2872eb4)
 #1 0x0000556c090a35de (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x28705de)
 #2 0x00007f71a4f4a3c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x153c0)
 #3 0x00007f71a49e918b raise /build/glibc-eX1tMB/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #4 0x00007f71a49c8859 abort /build/glibc-eX1tMB/glibc-2.31/stdlib/abort.c:81:7
 #5 0x00007f71a49c8729 get_sysdep_segment_value /build/glibc-eX1tMB/glibc-2.31/intl/loadmsgcat.c:509:8
 #6 0x00007f71a49c8729 _nl_load_domain /build/glibc-eX1tMB/glibc-2.31/intl/loadmsgcat.c:970:34
 #7 0x00007f71a49d9f36 (/lib/x86_64-linux-gnu/libc.so.6+0x36f36)
 #8 0x0000556c099580dd (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x31250dd)
 #9 0x0000556c09959cc0 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3126cc0)
#10 0x0000556c0995993e (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x312693e)
#11 0x0000556c0995993e (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x312693e)
#12 0x0000556c0995993e (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x312693e)
#13 0x0000556c0995993e (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x312693e)
#14 0x0000556c0995c144 clang::CodeGen::CGOpenMPRuntimeGPU::emitFunctionProlog(clang::CodeGen::CodeGenFunction&, clang::Decl const*) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3129144)
#15 0x0000556c09526be7 clang::CodeGen::CodeGenFunction::StartFunction(clang::GlobalDecl, clang::QualType, llvm::Function*, clang::CodeGen::CGFunctionInfo const&, clang::CodeGen::FunctionArgList const&, clang::SourceLocation, clang::SourceLocation) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cf3be7)
#16 0x0000556c095059a0 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cd29a0)
#17 0x0000556c09507296 clang::CodeGen::CodeGenFunction::GenerateOpenMPCapturedStmtFunction(clang::CapturedStmt const&, clang::SourceLocation) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cd4296)
#18 0x0000556c0991172d (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30de72d)
#19 0x0000556c09911e5b clang::CodeGen::CGOpenMPRuntime::emitTeamsOutlinedFunction(clang::OMPExecutableDirective const&, clang::VarDecl const*, llvm::omp::Directive, clang::CodeGen::RegionCodeGenTy const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30dee5b)
#20 0x0000556c09953fb2 clang::CodeGen::CGOpenMPRuntimeGPU::emitTeamsOutlinedFunction(clang::OMPExecutableDirective const&, clang::VarDecl const*, llvm::omp::Directive, clang::CodeGen::RegionCodeGenTy const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3120fb2)
#21 0x0000556c094eb3a3 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cb83a3)
#22 0x0000556c094ec009 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cb9009)
#23 0x0000556c098d8d1e clang::CodeGen::RegionCodeGenTy::operator()(clang::CodeGen::CodeGenFunction&) const (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30a5d1e)
#24 0x0000556c098d90dc (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30a60dc)
#25 0x0000556c0950828b clang::CodeGen::CodeGenFunction::GenerateOpenMPCapturedStmtFunction(clang::CapturedStmt const&, clang::SourceLocation) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2cd528b)
#26 0x0000556c0991f588 clang::CodeGen::CGOpenMPRuntime::emitTargetOutlinedFunctionHelper(clang::OMPExecutableDirective const&, llvm::StringRef, llvm::Function*&, llvm::Constant*&, bool, clang::CodeGen::RegionCodeGenTy const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30ec588)
#27 0x0000556c0994e94c clang::CodeGen::CGOpenMPRuntimeGPU::emitNonSPMDKernel(clang::OMPExecutableDirective const&, llvm::StringRef, llvm::Function*&, llvm::Constant*&, bool, clang::CodeGen::RegionCodeGenTy const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x311b94c)
#28 0x0000556c0994f4ef clang::CodeGen::CGOpenMPRuntimeGPU::emitTargetOutlinedFunction(clang::OMPExecutableDirective const&, llvm::StringRef, llvm::Function*&, llvm::Constant*&, bool, clang::CodeGen::RegionCodeGenTy const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x311c4ef)
#29 0x0000556c094c5bc9 clang::CodeGen::CodeGenFunction::EmitOMPTargetTeamsDistributeDeviceFunction(clang::CodeGen::CodeGenModule&, llvm::StringRef, clang::OMPTargetTeamsDistributeDirective const&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2c92bc9)
#30 0x0000556c098eadd3 clang::CodeGen::CGOpenMPRuntime::scanForTargetRegionsFunctions(clang::Stmt const*, llvm::StringRef) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30b7dd3)
#31 0x0000556c098eab3e clang::CodeGen::CGOpenMPRuntime::scanForTargetRegionsFunctions(clang::Stmt const*, llvm::StringRef) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30b7b3e)
#32 0x0000556c098eabef clang::CodeGen::CGOpenMPRuntime::scanForTargetRegionsFunctions(clang::Stmt const*, llvm::StringRef) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30b7bef)
#33 0x0000556c098eab3e clang::CodeGen::CGOpenMPRuntime::scanForTargetRegionsFunctions(clang::Stmt const*, llvm::StringRef) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30b7b3e)
#34 0x0000556c098eb0dd clang::CodeGen::CGOpenMPRuntime::emitTargetFunctions(clang::GlobalDecl) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x30b80dd)
#35 0x0000556c095839e7 clang::CodeGen::CodeGenModule::EmitGlobal(clang::GlobalDecl) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2d509e7)
#36 0x0000556c0958e5b2 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x2d5b5b2)
#37 0x0000556c0a2cf559 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3a9c559)
#38 0x0000556c0a2c1590 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3a8e590)
#39 0x0000556c0b043e74 clang::ParseAST(clang::Sema&, bool, bool) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x4810e74)
#40 0x0000556c0a2cc5d8 clang::CodeGenAction::ExecuteAction() (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3a995d8)
#41 0x0000556c09b9ff89 clang::FrontendAction::Execute() (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x336cf89)
#42 0x0000556c09b2d57e clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x32fa57e)
#43 0x0000556c09c7be23 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0x3448e23)
#44 0x0000556c0761da01 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0xdeaa01)
#45 0x0000556c076199c8 (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0xde69c8)
#46 0x0000556c07595f49 main (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0xd62f49)
#47 0x00007f71a49ca0b3 __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:342:3
#48 0x0000556c0761953e _start (/home/istvan/usr/lib/aomp_14.0-1/bin/clang-14+0xde653e)
clang-14: error: unable to execute command: Aborted (core dumped)
clang-14: error: clang frontend command failed due to signal (use -v to see invocation)
AOMP_STANDALONE_14.0-1 clang version 14.0.0 (https://github.com/radeonopencompute/llvm-project a4a503a2b65b37f4c8e4931d502cc6d53810b5f8)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/istvan/usr/lib/aomp_14.0-1/bin
clang-14: note: diagnostic msg: Error generating preprocessed source(s).
ERROR:  The following command failed with return code 254.
        /home/istvan/usr/lib/aomp_14.0-1/bin/clang  -g  -O3 -target x86_64-pc-linux-gnu -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906   -DLOMP -DL3D -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 -o  "/home/istvan/a.out" "omptest2.c"
carlobertolli commented 2 years ago

The rpm is probably packaging an older version. If you have a little time, I suggest you install from sources: https://github.com/ROCm-Developer-Tools/aomp/blob/aomp-dev/docs/SOURCEINSTALL.md

Otherwise, you can wait for a new rpm to be deployed.

reguly commented 2 years ago

I compiled the latest aomp-dev branch (4b0d880e9308916e25a7a42caf67e9b1030fb000) however, I see the exact same issue...

gregrodgers commented 2 years ago

Please try this on current release and close if resolved.

reguly commented 1 year ago

Just tested this with 16.0-1 (aompcc omptest.c -DLOMP -DL3D -O3 -fPIC -Wall -Wno-unused-result -Wno-unused-variable -ffast-math -g -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908) but still exact same ICE comes up

carlobertolli commented 1 year ago

Just tested with latest development branch. It gives a stack trace in the compiler, but the error is different from what originally reported:

clang-16: /home/cbertoll/git/aomp16/llvm-project/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp:250: void {anonymous}::CheckVarsEscapingDeclContext::markAsEscaped(const clang::ValueDecl*): Assertion `!VD->getType()->isVariablyModifiedType() && "Parameter captured by value with variably modified type"' failed.

doru1004 commented 1 year ago

I have reopened this since Carlo reported the error above. The fix for this latest error is here: https://reviews.llvm.org/D138614

ppanchad-amd commented 2 months ago

@reguly Can we close this ticket? Thanks!